We present a novel unsupervised learning approach to image landmark discovery by incorporating the inter-subject landmark consistencies on facial images. This is achieved via an inter-subject mapping module that transforms original subject landmarks based on an auxiliary subject-related structure. To recover from the transformed images back to the original subject, the landmark detector is forced to learn spatial locations that contain the consistent semantic meanings both for the paired intra-subject images and between the paired inter-subject images. Our proposed method is extensively evaluated on two public facial image datasets (MAFL, AFLW) with various settings. Experimental results indicate that our method can extract the consistent landmarks for both datasets and achieve better performances compared to the previous state-of-the-art methods quantitatively and qualitatively.
Enhanced Landmark Detection Model in Pelvic Fluoroscopy using 2D/3D Registration Loss
Department of Computer Science, Vanderbilt University（计算机科学系，范德堡大学）；Vanderbilt Institute for Surgery and Engineering（范德堡手术与工程研究院）；Department of Mathematics, University of California-Los Angeles（数学系，加州大学洛杉矶分校）；Vanderbilt Lab for Immersive AI Translation（范德堡沉浸式AI翻译实验室）；Department of Orthopedic Surgery, Vanderbilt University Medical Center（骨科手术系，范德堡大学医学中心）
Shandong University（山东大学）；Hong Kong University of Science and Technology（香港科技大学）
Mitral valve repair is a very difficult operation, often requiring experienced surgeons. The doctor will insert a prosthetic ring to aid in the restoration of heart function. The location of the prosthesis' sutures is critical. Obtaining and studying them during the procedure is a valuable learning experience for new surgeons. This paper proposes a landmark detection network for detecting sutures in endoscopic pictures, which solves the problem of a variable number of suture points in the images. Because there are two datasets, one from the simulated domain and the other from real intraoperati…
Unitary；VGG, University of Oxford（牛津大学VGG）；University of Edinburgh（爱丁堡大学）
Equivariance to random image transformations is an effective method to learn landmarks of object categories, such as the eyes and the nose in faces, without manual supervision. However, this method does not explicitly guarantee that the learned landmarks are consistent with changes between different instances of the same object, such as different facial identities. In this paper, we develop a new perspective on the equivariance approach by noting that dense landmark detectors can be interpreted as local image descriptors equipped with invariance to intra-category variations. We then propose a …
Shanghai Jiao Tong University（上海交通大学）；MediaTek USA Inc.（联发科技美国公司）
Video enhancement plays an important role in various video applications. In this paper, we propose a new intra-and-inter-constraint-based video enhancement approach aiming to 1) achieve high intra-frame quality of the entire picture where multiple region-of-interests (ROIs) can be adaptively and simultaneously enhanced, and 2) guarantee the inter-frame quality consistencies among video frames. We first analyze features from different ROIs and create a piecewise tone mapping curve for the entire frame such that the intra-frame quality of a frame can be enhanced. We further introduce new inter-f…
X-ray image guidance enables percutaneous alternatives to complex procedures. Unfortunately, the indirect view onto the anatomy in addition to projective simplification substantially increase the task-load for the surgeon. Additional 3D information such as knowledge of anatomical landmarks can benefit surgical decision making in complicated scenarios. Automatic detection of these landmarks in transmission imaging is challenging since image-domain features characteristic to a certain landmark change substantially depending on the viewing direction. Consequently and to the best of our knowledge,…
While image retrieval and instance recognition techniques are progressing rapidly, there is a need for challenging datasets to accurately measure their performance -- while posing novel challenges that are relevant for practical applications. We introduce the Google Landmarks Dataset v2 (GLDv2), a new benchmark for large-scale, fine-grained instance recognition and image retrieval in the domain of human-made and natural landmarks. GLDv2 is the largest such dataset to date by a large margin, including over 5M images and 200k distinct instance labels. Its test set consists of 118k images with gr…
An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion
University of Leeds（利兹大学）；Centre Hospitalier Universitaire de Clermont-Ferrand（克莱蒙费朗大学医院中心）；National University of Singapore (NUS)（新加坡国立大学）；National Center for Tumor Diseases (NCT/UCC Dresden)（德国国家肿瘤疾病中心（德累斯顿））；Fudan University（复旦大学）；University of Pennsylvania（宾夕法尼亚大学）
Deep neural networks have been widely used for feature learning in facial expression recognition systems. However, small datasets and large intra-class variability can lead to overfitting. In this paper, we propose a method which learns an optimized compact network topology for real-time facial expression recognition utilizing localized facial landmark features. Our method employs a spatio-temporal bilinear layer as backbone to capture the motion of facial landmarks during the execution of a facial expression effectively. Besides, it takes advantage of Monte Carlo Dropout to capture the model'…
From Pre- to Intra-operative MRI: Predicting Brain Shift in Temporal Lobe Resection for Epilepsy Surgery
Surgical and Interventional Engineering（外科与介入工程）；King’s College London（伦敦国王学院）；National Hospital for Neurology and Neurosurgery（神经外科医院）；NHS（国家健康服务）；Neurosurgery, King’s College Hospital（神经外科，伦敦国王学院医院）；Fondazione IRCCS Ca’ Granda（IRCCS Ca’ Granda基金会）；Ospedale Maggiore di Milano（米兰大医院）
Kyoto University（京都大学）；Tohoku University（东北大学）；Tokyo Institute of Technology（东京工业大学）；Nagoya University（名古屋大学）
In video-assisted thoracoscopic surgeries, successful procedures of nodule resection are highly dependent on the precise estimation of lung deformation between the inflated lung in the computed tomography (CT) images during preoperative planning and the deflated lung in the treatment views during surgery. Lungs in the pneumothorax state during surgery have a large volume change from normal lungs, making it difficult to build a mechanical model. The purpose of this study is to develop a deformation estimation method of the 3D surface of a deflated lung from a few partial observations. To estima…
The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance of the same surgical target structure. This can be viewed as a novel augmented reality approach, which we coined Hyperrealism in previous work. In this use case, it is of paramount importance to display objects like needles, sutures or instruments consistent in both domains while altering the style t…
PAL-Net: A Point-Wise CNN with Patch-Attention for 3D Facial Landmark Localization
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy.（电子、信息与生物工程系，米兰理工学院，米兰，意大利）；Department of Biomedical Sciences for Health, University of Milan, Milan, Italy.（健康生物医学科学系，米兰大学，米兰，意大利）；U.O. Laboratory of Applied Morphology, IRCCS Policlinico San Donato, 20097 San Donato Milanese, Italy.（应用形态学实验室，IRCCS圣丹诺多临床医院，意大利）；Department of Biomedical, Surgical and Dental Sciences, University of Milan, Milan, Italy.（生物医学、外科和牙科学系，米兰大学，米兰，意大利）
UAE: Universal Anatomical Embedding on Multi-modality Medical Images
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与工程学院，空天地海一体化大数据应用技术国家工程实验室）；Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Beijing United Family Hospital（北京和睦家医院）；The First Affiliated Hospital, Zhejiang University（浙江大学医学院附属第一医院）；Peking Union Medical College Hospital（北京协和医院）
The various speech sounds of a language are obtained by varying the shape and position of the articulators surrounding the vocal tract. Analyzing their variations is crucial for understanding speech production, diagnosing speech disorders and planning therapy. Identifying key anatomical landmarks of these structures on medical images is a pre-requisite for any quantitative analysis and the rising amount of data generated in the field calls for an automatic solution. The challenge lies in the high inter- and intra-speaker variability, the mutual interaction between the articulators and the mode…
Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization
School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；Defense Innovation Institute, Academy of Military Sciences（军事科学院国防创新研究院）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）；High-tech Institute of Qingzhou（青州高新技术研究院）
University College London（伦敦大学学院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Allen Institute for Brain Science（艾伦脑科学研究所）；Massachusetts Institute of Technology（麻省理工学院）
Nonlinear registration of 2D histological sections with corresponding slices of MRI data is a critical step of 3D histology reconstruction. This task is difficult due to the large differences in image contrast and resolution, as well as the complex nonrigid distortions produced when sectioning the sample and mounting it on the glass slide. It has been shown in brain MRI registration that better spatial alignment across modalities can be obtained by synthesizing one modality from the other and then using intra-modality registration metrics, rather than by using mutual information (MI) as metric…
Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃尔兰根-纽伦堡大学）；Siemens Healthcare GmbH（西门子医疗保健有限公司）；Hochschule Furtwangen（富特旺根应用技术大学）；BG Trauma Center Ludwigshafen（路德维希港BG创伤中心）
X-ray based measurement and guidance are commonly used tools in orthopaedic surgery to facilitate a minimally invasive workflow. Typically, a surgical planning is first performed using knowledge of bone morphology and anatomical landmarks. Information about bone location then serves as a prior for registration during overlay of the planning on intra-operative X-ray images. Performing these steps manually however is prone to intra-rater/inter-rater variability and increases task complexity for the surgeon. To remedy these issues, we propose an automatic framework for planning and subsequent ove…
University College London（伦敦大学学院）；University of Oxford（牛津大学）；University College London Hospital NHS Foundation Trust（伦敦大学学院医院国民保健服务基金会信托）；Stanford University（斯坦福大学）
Image registration is useful for quantifying morphological changes in longitudinal MR images from prostate cancer patients. This paper describes a development in improving the learning-based registration algorithms, for this challenging clinical application often with highly variable yet limited training data. First, we report that the latent space can be clustered into a much lower dimensional space than that commonly found as bottleneck features at the deep layer of a trained registration network. Based on this observation, we propose a hierarchical quantization method, discretizing the lear…
Improving Automatic Fetal Biometry Measurement with Swoosh Activation Function
University of Sydney（悉尼大学）；James Cook University（詹姆斯库克大学）；Central Queensland University（中央昆士兰大学）；Liverpool Hospital（利物浦医院）；Sydney Medical School Nepean（悉尼医学学院尼皮恩分校）
Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Alibaba Group（阿里巴巴集团）；Southeast University（东南大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Institute of Medical Informatics, Universität zu Lübeck（吕贝克大学医学信息学研究所）
Nonlinear image registration continues to be a fundamentally important tool in medical image analysis. Diagnostic tasks, image-guided surgery and radiotherapy as well as motion analysis all rely heavily on accurate intra-patient alignment. Furthermore, inter-patient registration enables atlas-based segmentation or landmark localisation and shape analysis. When labelled scans are scarce and anatomical differences large, conventional registration has often remained superior to deep learning methods that have so far mainly dealt with relatively small or low-complexity deformations. We address thi…
University of Sussex（萨塞克斯大学）；Polytechnique Montreal（蒙特利尔理工学院）；Cardiff University（卡迪夫大学）；University College London（伦敦大学学院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
Nonlinear inter-modality registration is often challenging due to the lack of objective functions that are good proxies for alignment. Here we propose a synthesis-by-registration method to convert this problem into an easier intra-modality task. We introduce a registration loss for weakly supervised image translation between domains that does not require perfectly aligned training data. This loss capitalises on a registration U-Net with frozen weights, to drive a synthesis CNN towards the desired translation. We complement this loss with a structure preserving constraint based on contrastive l…
