Echocardiography has become routinely used in the diagnosis of cardiomyopathy and abnormal cardiac blood flow. However, manually measuring myocardial motion and cardiac blood flow from echocardiogram is time-consuming and error-prone. Computer algorithms that can automatically track and quantify myocardial motion and cardiac blood flow are highly sought after, but have not been very successful due to noise and high variability of echocardiography. In this work, we propose a neural multi-scale self-supervised registration (NMSR) method for automated myocardial and cardiac blood flow dense tracking. NMSR incorporates two novel components: 1) utilizing a deep neural net to parameterize the velocity field between two image frames, and 2) optimizing the parameters of the neural net in a sequential multi-scale fashion to account for large variations within the velocity field. Experiments demonstrate that NMSR yields significantly better registration accuracy than state-of-the-art methods, such as advanced normalization tools (ANTs) and VoxelMorph, for both myocardial and cardiac blood flow dense tracking. Our approach promises to provide a fully automated method for fast and accurate analyses of echocardiograms.
Markerless Tracking-Based Registration for Medical Image Motion Correction
Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg（生物医学工程人工智能系，埃尔朗根-纽伦堡大学）；Department of Communication Sciences and Disorders, East Carolina University（通讯科学与障碍系，东卡罗来纳大学）；Department of Otolaryngology–Head and Neck Surgery, Louisiana State University Health Sciences Center（耳鼻喉科与头颈外科系，路易斯安那州立大学健康科学中心）；Department of Radiology, FAU Erlangen-Nürnberg（放射学系，埃尔朗根-纽伦堡大学）；Department of Communication Sciences and Disorders, Louisiana State University（通讯科学与障碍系，路易斯安那州立大学）
University of Freiburg（弗莱堡大学）；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））
Deformable image registration is a very important field of research in medical imaging. Recently multiple deep learning approaches were published in this area showing promising results. However, drawbacks of deep learning methods are the need for a large amount of training datasets and their inability to register unseen images different from the training datasets. One shot learning comes without the need of large training datasets and has already been proven to be applicable to 3D data. In this work we present a one shot registration approach for periodic motion tracking in 3D and 4D datasets.…
Uppsala University（乌普萨拉大学）；Elekta Instrument AB（医科达仪器有限公司）
Our anatomy is in constant motion. With modern MR imaging it is possible to record this motion in real-time during an ongoing radiation therapy session. In this paper we present an image registration method that exploits the sequential nature of 2D MR images to estimate the corresponding displacement field. The method employs several discriminative correlation filters that independently track specific points. Together with a sparse-to-dense interpolation scheme we can then estimate of the displacement field. The discriminative correlation filters are trained online, and our method is modality …
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）；Radboud UMC（拉德堡德大学医学中心）；University of Lübeck（吕贝克大学）
While deep learning has achieved significant advances in accuracy for medical image segmentation, its benefits for deformable image registration have so far remained limited to reduced computation times. Previous work has either focused on replacing the iterative optimization of distance and smoothness terms with CNN-layers or using supervised approaches driven by labels. Our method is the first to combine the complementary strengths of global semantic information (represented by segmentation labels) and local distance metrics that help align surrounding structures. We demonstrate significant …
Nanjing University of Science and Technology（南京理工大学）
Learning robust feature matching between the template and search area is crucial for 3D Siamese tracking. The core of Siamese feature matching is how to assign high feature similarity on the corresponding points between the template and search area for precise object localization. In this paper, we propose a novel point cloud registration-driven Siamese tracking framework, with the intuition that spatially aligned corresponding points (via 3D registration) tend to achieve consistent feature representations. Specifically, our method consists of two modules, including a tracking-specific nonloca…
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
WarpEM: Dynamic Time Warping for Accurate Catheter Registration in EM-guided Procedures
Technical University of Munich（慕尼黑工业大学）；German Heart Center Munich（慕尼黑德国心脏中心）；German Centre for Cardiovascular Research（德国心血管研究中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
Improve Myocardial Strain Estimation based on Deformable Groupwise Registration with a Locally Low-Rank Dissimilarity Metric
Shanghai Jiao Tong University（上海交通大学）；School of Biomedical Engineering（生物医学工程学院）；National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy（国家医学诊疗先进磁共振技术工程研究中心）；Renji Hospital, School of Medicine, Shanghai Jiao Tong University（上海交通大学医学院附属仁济医院）
This paper adapts a popular image quality measure called structural similarity for high precision registration based tracking while also introducing a simpler and faster variant of the same. Further, these are evaluated comprehensively against existing measures using a unified approach to study registration based trackers that decomposes them into three constituent sub modules - appearance model, state space model and search method. Several popular trackers in literature are broken down using this method so that their contributions - as of this paper - are shown to be limited to only one or tw…
This paper presents a modular, extensible and highly efficient open source framework for registration based tracking called Modular Tracking Framework (MTF). Targeted at robotics applications, it is implemented entirely in C++ and designed from the ground up to easily integrate with systems that support any of several major vision and robotics libraries including OpenCV, ROS, ViSP and Eigen. It implements more methods, is faster, and more precise than other existing systems. Further, the theoretical basis for its design is a new way to conceptualize registration based trackers that decomposes …
Dense real-time tracking and mapping from RGB-D images is an important tool for many robotic applications, such as navigation and manipulation. The recently presented Directional Truncated Signed Distance Function (DTSDF) is an augmentation of the regular TSDF that shows potential for more coherent maps and improved tracking performance. In this work, we present methods for rendering depth- and color images from the DTSDF, making it a true drop-in replacement for the regular TSDF in established trackers. We evaluate the algorithm on well-established datasets and observe that our method improve…
University of Rochester（罗切斯特大学）；University of Maryland, College Park（马里兰大学帕克分校）；Rutgers University（罗格斯大学）；Z2W Corporation（Z2W公司）；Chinese Academy of Sciences（中国科学院）；Peng Cheng Laboratory（鹏城实验室）
We propose to tackle the problem of multiview 2D/3D rigid registration for intervention via a Point-Of-Interest Network for Tracking and Triangulation ($\text{POINT}^2$). $\text{POINT}^2$ learns to establish 2D point-to-point correspondences between the pre- and intra-intervention images by tracking a set of random POIs. The 3D pose of the pre-intervention volume is then estimated through a triangulation layer. In $\text{POINT}^2$, the unified framework of the POI tracker and the triangulation layer enables learning informative 2D features and estimating 3D pose jointly. In contrast to existin…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Temple University（天普大学）
There has been a long pursuit for precise and reproducible glomerular quantification on renal pathology to leverage both research and practice. When digitizing the biopsy tissue samples using whole slide imaging (WSI), a set of serial sections from the same tissue can be acquired as a stack of images, similar to frames in a video. In radiology, the stack of images (e.g., computed tomography) are naturally used to provide 3D context for organs, tissues, and tumors. In pathology, it is appealing to do a similar 3D assessment. However, the 3D identification and association of large-scale glomerul…
Imperial College London（帝国理工学院）；University of Edinburgh（爱丁堡大学）
Image registration is an ill-posed inverse problem which often requires regularisation on the solution space. In contrast to most of the current approaches which impose explicit regularisation terms such as smoothness, in this paper we propose a novel method that can implicitly learn biomechanics-informed regularisation. Such an approach can incorporate application-specific prior knowledge into deep learning based registration. Particularly, the proposed biomechanics-informed regularisation leverages a variational autoencoder (VAE) to learn a manifold for biomechanically plausible deformations…
University of Utah（犹他大学）；University of Maryland, School of Medicine（马里兰大学医学院）
Radiation therapy presents a need for dynamic tracking of a target tumor volume. Fiducial markers such as implanted gold seeds have been used to gate radiation delivery but the markers are invasive and gating significantly increases treatment time. Pretreatment acquisition of a respiratory correlated 4DCT allows for determination of accurate motion tracking which is useful in treatment planning. We design a patient-specific motion subspace and a deep convolutional neural network to recover anatomical positions from a single fluoroscopic projection in real-time. We use this deep network to appr…
Images taken at different times or positions undergo transformations such as rotation, scaling, skewing, and more. The process of aligning different images which have undergone transformations can be done via registration. Registration is desirable when analyzing time-series data for tracking, averaging, or differential diagnoses of diseases. Efficient registration methods exist for rigid (including linear or affine) transformations; however, for non-rigid (also known as non-affine) transformations, current methods are computationally expensive and time-consuming. In this report, I will explor…
Yale University（耶鲁大学）；University of Pennsylvania（宾夕法尼亚大学）
Augmented reality has the potential to improve operating room workflow by allowing physicians to "see" inside a patient through the projection of imaging directly onto the surgical field. For this to be useful the acquired imaging must be quickly and accurately registered with patient and the registration must be maintained. Here we describe a method for projecting a CT scan with Microsoft Hololens and then aligning that projection to a set of fiduciary markers. Radio-opaque stickers with unique QR-codes are placed on an object prior to acquiring a CT scan. The location of the markers in the C…
Longitudinal tracking of multiple sclerosis lesions in the spinal cord: A validation study
Polytechnique Montreal（蒙特利尔理工学院）；TUM University Hospital（慕尼黑工业大学附属医院）；University of British Columbia（不列颠哥伦比亚大学）；University of Calgary（卡尔加里大学）；University of Toronto（多伦多大学）；Université de Montréal（蒙特利尔大学）；University of Alberta（阿尔伯塔大学）
With superiorities on low cost, portability, and free of radiation, echocardiogram is a widely used imaging modality for left ventricle (LV) function quantification. However, automatic LV segmentation and motion tracking is still a challenging task. In addition to fuzzy border definition, low contrast, and abounding artifacts on typical ultrasound images, the shape and size of the LV change significantly in a cardiac cycle. In this work, we propose a temporal affine network (TAN) to perform image analysis in a warped image space, where the shape and size variations due to the cardiac motion as…
Registration is a fundamental task in medical image analysis which can be applied to several tasks including image segmentation, intra-operative tracking, multi-modal image alignment, and motion analysis. Popular registration tools such as ANTs and NiftyReg optimize an objective function for each pair of images from scratch which is time-consuming for large images with complicated deformation. Facilitated by the rapid progress of deep learning, learning-based approaches such as VoxelMorph have been emerging for image registration. These approaches can achieve competitive performance in a fract…
King’s College London（伦敦国王学院）；Siemens Healthineers（西门子医疗）
Registering CT images of the chest is a crucial step for several tasks such as disease progression tracking or surgical planning. It is also a challenging step because of the heterogeneous content of the human abdomen which implies complex deformations. In this work, we focus on accurately registering a subset of organs of interest. We register organ surface point clouds, as may typically be extracted from an automatic segmentation pipeline, by expanding the Bayesian Coherent Point Drift algorithm (BCPD). We introduce MO-BCPD, a multi-organ version of the BCPD algorithm which explicitly models…
Continuous 3D Myocardial Motion Tracking via Echocardiography
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；Medical School, Nanjing University（南京大学医学院）；Zhongshan Hospital, Fudan University（复旦大学中山医院）；College of Optical Sciences, University of Arizona（亚利桑那大学光学科学学院）
GLIDE-Reg: Global-to-Local Deformable Registration Using Co-Optimized Foundation and Handcrafted Features
1 Medical \& Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, USA 2 Magnetic Resonance Research Labs, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, USA 3 Department of Radiation Oncology, David Geffen School of Medicine at UCLA, Los Angeles, USA
SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（香港中文大学（深圳）与中国科学院深圳先进技术研究院联合机构））；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapy
Department of Computer Science and Software Engineering, Concordia University（计算机科学与软件工程系，康科迪亚大学）；Department of Electrical and Computer Engineering, Concordia University（电气与计算机工程系，康科迪亚大学）；Elekta Ltd.（Elekta公司）；Diagnos Medical Systems（Diagnos医疗系统）
Rutgers University（罗格斯大学）；New York University School of Medicine（纽约大学医学院）；NVIDIA（英伟达）；SenseBrain；Shanghai AI Laboratory（上海人工智能实验室）
Cardiac tagging magnetic resonance imaging (t-MRI) is the gold standard for regional myocardium deformation and cardiac strain estimation. However, this technique has not been widely used in clinical diagnosis, as a result of the difficulty of motion tracking encountered with t-MRI images. In this paper, we propose a novel deep learning-based fully unsupervised method for in vivo motion tracking on t-MRI images. We first estimate the motion field (INF) between any two consecutive t-MRI frames by a bi-directional generative diffeomorphic registration neural network. Using this result, we then e…
University of Houston（休斯顿大学）；Boston University（波士顿大学）；Rice University（莱斯大学）
Our work targets automated analysis to quantify the growth dynamics of a population of bacilliform bacteria. We propose an innovative approach to frame-sequence tracking of deformable-cell motion by the automated minimization of a new, specific cost functional. This minimization is implemented by dedicated Boltzmann machines (stochastic recurrent neural networks). Automated detection of cell divisions is handled similarly by successive minimizations of two cost functions, alternating the identification of children pairs and parent identification. We validate the proposed automatic cell trackin…
Bayesian In Vivo Tracking of Synapses using Joint Poisson Deconvolution and Diffeomorphic Registration
Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Department of Neuroscience, Johns Hopkins University（约翰霍普金斯大学神经科学系）；Kavli Neuroscience Discovery Institute, Johns Hopkins University（约翰霍普金斯大学Kavli神经科学发现研究所）；Data Science and AI Institute, Johns Hopkins University（约翰霍普金斯大学数据科学与人工智能研究所）；Department of Applied Mathematics and Statistics, Johns Hopkins University（约翰霍普金斯大学应用数学与统计学系）
SLAM assisted 3D tracking system for laparoscopic surgery
United Imaging Research Institute of Intelligent Imaging（联影智能影像研究院）；University of Michigan（密歇根大学）；United Imaging（联影医疗）；United Imaging of Surgery（联影外科研究院）
University of Technology Sydney（悉尼科技大学）；Imperial College London（帝国理工学院）
This work addresses the issue of motion compensation and pattern tracking in event camera data. An event camera generates asynchronous streams of events triggered independently by each of the pixels upon changes in the observed intensity. Providing great advantages in low-light and rapid-motion scenarios, such unconventional data present significant research challenges as traditional vision algorithms are not directly applicable to this sensing modality. The proposed method decomposes the tracking problem into a local SE(2) motion-compensation step followed by a homography registration of smal…
Deformable surface tracking from monocular images is well-known to be under-constrained. Occlusions often make the task even more challenging, and can result in failure if the surface is not sufficiently textured. In this work, we explicitly address the problem of 3D reconstruction of poorly textured, occluded surfaces, proposing a framework based on a template-matching approach that scales dense robust features by a relevancy score. Our approach is extensively compared to current methods employing both local feature matching and dense template alignment. We test on standard datasets as well a…
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术院（KAIST））
Point cloud registration is a fundamental task in many applications such as localization, mapping, tracking, and reconstruction. Successful registration relies on extracting robust and discriminative geometric features. Though existing learning based methods require high computing capacity for processing a large number of raw points at the same time, computational capacity limitation is not an issue thanks to the powerful parallel computing process using GPU. In this paper, we introduce a framework that efficiently and economically extracts dense features using a graph attention network for po…
Princeton University（普林斯顿大学）；Lewis-Sigler Institute for Integrative Genomics, Princeton University（普林斯顿大学刘易斯-西格勒综合基因组学研究所）；Princeton Neuroscience Institute, Princeton University（普林斯顿大学普林斯顿神经科学研究所）；Tufts University School of Medicine（塔夫茨大学医学院）
Advances in optical neuroimaging techniques now allow neural activity to be recorded with cellular resolution in awake and behaving animals. Brain motion in these recordings pose a unique challenge. The location of individual neurons must be tracked in 3D over time to accurately extract single neuron activity traces. Recordings from small invertebrates like C. elegans are especially challenging because they undergo very large brain motion and deformation during animal movement. Here we present an automated computer vision pipeline to reliably track populations of neurons with single neuron res…
University of Pennsylvania（宾夕法尼亚大学）；University of Minnesota（明尼苏达大学）
This paper presents a semi-supervised learning framework to train a keypoint detector using multiview image streams given the limited labeled data (typically $<$4\%). We leverage the complementary relationship between multiview geometry and visual tracking to provide three types of supervisionary signals to utilize the unlabeled data: (1) keypoint detection in one view can be supervised by other views via the epipolar geometry; (2) a keypoint moves smoothly over time where its optical flow can be used to temporally supervise consecutive image frames to each other; (3) visible keypoint in one v…
Exploiting Longitudinal Context in Clinician-Verified Interactive Lesion Tracking
German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Germany（德国癌症研究中心（DKFZ）海德堡，医学图像计算部，德国）；Faculty of Mathematics and Computer Science, Heidelberg University, Germany（海德堡大学数学与计算机科学学院，德国）；HIDSS4Health -- Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany（HIDSS4Health——海德堡信息与数据科学健康学校，卡尔斯鲁厄/海德堡，德国）；Medical Faculty, Heidelberg University, Germany（海德堡大学医学学院，德国）；University Hospital Brandenburg an der Havel, Brandenburg Medical School Theodor Fontane, Germany（勃兰登堡运河大学医院，布兰登堡泰奥多尔·冯·_fontane医学学校，德国）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Germany（放射肿瘤科模式分析与学习组，海德堡大学医院，德国）
Methods tackling multi-object tracking need to estimate the number of targets in the sensing area as well as to estimate their continuous state. While the majority of existing methods focus on data association, precise state (3D pose) estimation is often only coarsely estimated by approximating targets with centroids or (3D) bounding boxes. However, in automotive scenarios, motion perception of surrounding agents is critical and inaccuracies in the vehicle close-range can have catastrophic consequences. In this work, we focus on precise 3D track state estimation and propose a learning-based ap…
WarpPINN: Cine-MR image registration with physics-informed neural networks
University of Bath（巴斯大学）；Pontificia Universidad Católica de Valparaíso（瓦尔帕莱索天主大学）；Millennium Institute for Intelligent Healthcare Engineering（千年智能医疗工程研究所）；Pontificia Universidad Católica de Chile（智利天主大学）
GuidedMorph: Two-Stage Deformable Registration for Breast MRI
Department of Electrical and Computer Engineering, Duke University（电子工程系，杜克大学）；Department of Biostatistics & Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（健康结果与生物医学信息学系，医学院，佛罗里达大学）
Object triangulation, 3-D object tracking, feature correspondence, and camera calibration are key problems for estimation from camera networks. This paper addresses these problems within a unified Bayesian framework for joint multi-object tracking and sensor registration. Given that using standard filtering approaches for state estimation from cameras is problematic, an alternative parametrisation is exploited, called disparity space. The disparity space-based approach for triangulation and object tracking is shown to be more effective than non-linear versions of the Kalman filter and particle…
Automatic Multi-View X-Ray/CT Registration Using Bone Substructure Contours
University Children’s Hospital Zürich（苏黎世儿童医院大学医院）；University Hospital Balgrist, University of Zurich（苏黎世大学医院巴尔格里斯分院）；Department of Orthopedic Surgery（骨科部门）
University of Edinburgh（爱丁堡大学）；Fudan University（复旦大学）；Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
Myocardial motion and deformation are rich descriptors that characterize cardiac function. Image registration, as the most commonly used technique for myocardial motion tracking, is an ill-posed inverse problem which often requires prior assumptions on the solution space. In contrast to most existing approaches which impose explicit generic regularization such as smoothness, in this work we propose a novel method that can implicitly learn an application-specific biomechanics-informed prior and embed it into a neural network-parameterized transformation model. Particularly, the proposed method …
