This paper adapts a popular image quality measure called structural similarity for high precision registration based tracking while also introducing a simpler and faster variant of the same. Further, these are evaluated comprehensively against existing measures using a unified approach to study registration based trackers that decomposes them into three constituent sub modules - appearance model, state space model and search method. Several popular trackers in literature are broken down using this method so that their contributions - as of this paper - are shown to be limited to only one or two of these submodules. An open source tracking framework is made available that follows this decomposition closely through extensive use of generic programming. It is used to perform all experiments on four publicly available datasets so the results are easily reproducible. This framework provides a convenient interface to plug in a new method for any sub module and combine it with existing methods for the other two. It can also serve as a fast and flexible solution for practical tracking needs due to its highly efficient implementation.
University of Freiburg（弗莱堡大学）；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））
Deformable image registration is a very important field of research in medical imaging. Recently multiple deep learning approaches were published in this area showing promising results. However, drawbacks of deep learning methods are the need for a large amount of training datasets and their inability to register unseen images different from the training datasets. One shot learning comes without the need of large training datasets and has already been proven to be applicable to 3D data. In this work we present a one shot registration approach for periodic motion tracking in 3D and 4D datasets.…
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）；Radboud UMC（拉德堡德大学医学中心）；University of Lübeck（吕贝克大学）
While deep learning has achieved significant advances in accuracy for medical image segmentation, its benefits for deformable image registration have so far remained limited to reduced computation times. Previous work has either focused on replacing the iterative optimization of distance and smoothness terms with CNN-layers or using supervised approaches driven by labels. Our method is the first to combine the complementary strengths of global semantic information (represented by segmentation labels) and local distance metrics that help align surrounding structures. We demonstrate significant …
WarpEM: Dynamic Time Warping for Accurate Catheter Registration in EM-guided Procedures
Technical University of Munich（慕尼黑工业大学）；German Heart Center Munich（慕尼黑德国心脏中心）；German Centre for Cardiovascular Research（德国心血管研究中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
University of Rochester（罗切斯特大学）；University of Maryland, College Park（马里兰大学帕克分校）；Rutgers University（罗格斯大学）；Z2W Corporation（Z2W公司）；Chinese Academy of Sciences（中国科学院）；Peng Cheng Laboratory（鹏城实验室）
We propose to tackle the problem of multiview 2D/3D rigid registration for intervention via a Point-Of-Interest Network for Tracking and Triangulation ($\text{POINT}^2$). $\text{POINT}^2$ learns to establish 2D point-to-point correspondences between the pre- and intra-intervention images by tracking a set of random POIs. The 3D pose of the pre-intervention volume is then estimated through a triangulation layer. In $\text{POINT}^2$, the unified framework of the POI tracker and the triangulation layer enables learning informative 2D features and estimating 3D pose jointly. In contrast to existin…
Nanjing University of Science and Technology（南京理工大学）
Learning robust feature matching between the template and search area is crucial for 3D Siamese tracking. The core of Siamese feature matching is how to assign high feature similarity on the corresponding points between the template and search area for precise object localization. In this paper, we propose a novel point cloud registration-driven Siamese tracking framework, with the intuition that spatially aligned corresponding points (via 3D registration) tend to achieve consistent feature representations. Specifically, our method consists of two modules, including a tracking-specific nonloca…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Temple University（天普大学）
There has been a long pursuit for precise and reproducible glomerular quantification on renal pathology to leverage both research and practice. When digitizing the biopsy tissue samples using whole slide imaging (WSI), a set of serial sections from the same tissue can be acquired as a stack of images, similar to frames in a video. In radiology, the stack of images (e.g., computed tomography) are naturally used to provide 3D context for organs, tissues, and tumors. In pathology, it is appealing to do a similar 3D assessment. However, the 3D identification and association of large-scale glomerul…
CASIA（中国科学院自动化研究所）；University of Oxford（牛津大学）；FiveAI
In this paper we illustrate how to perform both visual object tracking and semi-supervised video object segmentation, in real-time, with a single simple approach. Our method, dubbed SiamMask, improves the offline training procedure of popular fully-convolutional Siamese approaches for object tracking by augmenting their loss with a binary segmentation task. Once trained, SiamMask solely relies on a single bounding box initialisation and operates online, producing class-agnostic object segmentation masks and rotated bounding boxes at 55 frames per second. Despite its simplicity, versatility and…
Tencent AI Lab（腾讯人工智能实验室）；Tencent Robotics X（腾讯Robotics X实验室）
Robust and accurate planar tracking over a whole video sequence is vitally important for many vision applications. The key to planar object tracking is to find object correspondences, modeled by homography, between the reference image and the tracked image. Existing methods tend to obtain wrong correspondences with changing appearance variations, camera-object relative motions and occlusions. To alleviate this problem, we present a unified convolutional neural network (CNN) model that jointly considers homography, visibility, and confidence. First, we introduce correlation blocks that explicit…
OmniTracker: Unifying Object Tracking by Tracking-with-Detection
Shanghai Key Lab of Intelligent Information Processing and School of Computer Science, Fudan University（上海智能信息处理关键实验室和复旦大学计算机学院）；Microsoft Research, Redmond（微软研究院（红mond））；Microsoft Research, Asia（微软亚洲研究院）
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Tracking objects over long videos effectively means solving a spectrum of problems, from short-term association for un-occluded objects to long-term association for objects that are occluded and then reappear in the scene. Methods tackling these two tasks are often disjoint and crafted for specific scenarios, and top-performing approaches are often a mix of techniques, which yields engineering-heavy solutions that lack generality. In this work, we question the need for hybrid approaches and introduce SUSHI, a unified and scalable multi-object tracker. Our approach processes long clips by split…
Markerless Tracking-Based Registration for Medical Image Motion Correction
Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg（生物医学工程人工智能系，埃尔朗根-纽伦堡大学）；Department of Communication Sciences and Disorders, East Carolina University（通讯科学与障碍系，东卡罗来纳大学）；Department of Otolaryngology–Head and Neck Surgery, Louisiana State University Health Sciences Center（耳鼻喉科与头颈外科系，路易斯安那州立大学健康科学中心）；Department of Radiology, FAU Erlangen-Nürnberg（放射学系，埃尔朗根-纽伦堡大学）；Department of Communication Sciences and Disorders, Louisiana State University（通讯科学与障碍系，路易斯安那州立大学）
Yale University（耶鲁大学）；University of Pennsylvania（宾夕法尼亚大学）
Augmented reality has the potential to improve operating room workflow by allowing physicians to "see" inside a patient through the projection of imaging directly onto the surgical field. For this to be useful the acquired imaging must be quickly and accurately registered with patient and the registration must be maintained. Here we describe a method for projecting a CT scan with Microsoft Hololens and then aligning that projection to a set of fiduciary markers. Radio-opaque stickers with unique QR-codes are placed on an object prior to acquiring a CT scan. The location of the markers in the C…
Multiple existing benchmarks involve tracking and segmenting objects in video e.g., Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS), but there is little interaction between them due to the use of disparate benchmark datasets and metrics (e.g. J&F, mAP, sMOTSA). As a result, published works usually target a particular benchmark, and are not easily comparable to each another. We believe that the development of generalized methods that can tackle multiple tasks requires greater cohesion among these research sub-communities. In this paper, we aim to facilitate thi…
This paper presents a modular, extensible and highly efficient open source framework for registration based tracking called Modular Tracking Framework (MTF). Targeted at robotics applications, it is implemented entirely in C++ and designed from the ground up to easily integrate with systems that support any of several major vision and robotics libraries including OpenCV, ROS, ViSP and Eigen. It implements more methods, is faster, and more precise than other existing systems. Further, the theoretical basis for its design is a new way to conceptualize registration based trackers that decomposes …
Dense real-time tracking and mapping from RGB-D images is an important tool for many robotic applications, such as navigation and manipulation. The recently presented Directional Truncated Signed Distance Function (DTSDF) is an augmentation of the regular TSDF that shows potential for more coherent maps and improved tracking performance. In this work, we present methods for rendering depth- and color images from the DTSDF, making it a true drop-in replacement for the regular TSDF in established trackers. We evaluate the algorithm on well-established datasets and observe that our method improve…
Longitudinal tracking of multiple sclerosis lesions in the spinal cord: A validation study
Polytechnique Montreal（蒙特利尔理工学院）；TUM University Hospital（慕尼黑工业大学附属医院）；University of British Columbia（不列颠哥伦比亚大学）；University of Calgary（卡尔加里大学）；University of Toronto（多伦多大学）；Université de Montréal（蒙特利尔大学）；University of Alberta（阿尔伯塔大学）
Improve Myocardial Strain Estimation based on Deformable Groupwise Registration with a Locally Low-Rank Dissimilarity Metric
Shanghai Jiao Tong University（上海交通大学）；School of Biomedical Engineering（生物医学工程学院）；National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy（国家医学诊疗先进磁共振技术工程研究中心）；Renji Hospital, School of Medicine, Shanghai Jiao Tong University（上海交通大学医学院附属仁济医院）
OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning
Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理重点实验室）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Engineering Research Center of AI & Robotics（上海AI与机器人工程研究中心）；Academy for Engineering & Technology, Fudan University（复旦大学工程技术研究院）；Engineering Research Center of AI & Robotics, Ministry of Education（教育部AI与机器人工程研究中心）
Unifying Visual and Vision-Language Tracking via Contrastive Learning
Deep Space Exploration Laboratory（深空探测实验室）；University of Science and Technology of China（中国科学技术大学）；Intelligent Science Technology Academy of CASIC（中国航天科工集团智能科学技术研究院）
NVIDIA（英伟达）；Lenovo Research（联想研究院）；Piedmont Heart Institute（皮埃蒙特心脏研究所）；Tencent Medical AI Lab（腾讯医疗人工智能实验室）；University of California, Irvine（加州大学欧文分校）
Echocardiography has become routinely used in the diagnosis of cardiomyopathy and abnormal cardiac blood flow. However, manually measuring myocardial motion and cardiac blood flow from echocardiogram is time-consuming and error-prone. Computer algorithms that can automatically track and quantify myocardial motion and cardiac blood flow are highly sought after, but have not been very successful due to noise and high variability of echocardiography. In this work, we propose a neural multi-scale self-supervised registration (NMSR) method for automated myocardial and cardiac blood flow dense track…
Uppsala University（乌普萨拉大学）；Elekta Instrument AB（医科达仪器有限公司）
Our anatomy is in constant motion. With modern MR imaging it is possible to record this motion in real-time during an ongoing radiation therapy session. In this paper we present an image registration method that exploits the sequential nature of 2D MR images to estimate the corresponding displacement field. The method employs several discriminative correlation filters that independently track specific points. Together with a sparse-to-dense interpolation scheme we can then estimate of the displacement field. The discriminative correlation filters are trained online, and our method is modality …
Bayesian In Vivo Tracking of Synapses using Joint Poisson Deconvolution and Diffeomorphic Registration
Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Department of Neuroscience, Johns Hopkins University（约翰霍普金斯大学神经科学系）；Kavli Neuroscience Discovery Institute, Johns Hopkins University（约翰霍普金斯大学Kavli神经科学发现研究所）；Data Science and AI Institute, Johns Hopkins University（约翰霍普金斯大学数据科学与人工智能研究所）；Department of Applied Mathematics and Statistics, Johns Hopkins University（约翰霍普金斯大学应用数学与统计学系）
Imperial College London（帝国理工学院）；University of Edinburgh（爱丁堡大学）
Image registration is an ill-posed inverse problem which often requires regularisation on the solution space. In contrast to most of the current approaches which impose explicit regularisation terms such as smoothness, in this paper we propose a novel method that can implicitly learn biomechanics-informed regularisation. Such an approach can incorporate application-specific prior knowledge into deep learning based registration. Particularly, the proposed biomechanics-informed regularisation leverages a variational autoencoder (VAE) to learn a manifold for biomechanically plausible deformations…
We propose a minimal solution for the similarity registration (rigid pose and scale) between two sets of 3D lines, and also between a set of co-planar points and a set of 3D lines. The first problem is solved up to 8 discrete solutions with a minimum of 2 line-line correspondences, while the second is solved up to 4 discrete solutions using 4 point-line correspondences. We use these algorithms to perform the extrinsic calibration between a pose tracking sensor and a 2D/3D ultrasound (US) curvilinear probe using a tracked needle as calibration target. The needle is tracked as a 3D line, and is …
University of Utah（犹他大学）；University of Maryland, School of Medicine（马里兰大学医学院）
Radiation therapy presents a need for dynamic tracking of a target tumor volume. Fiducial markers such as implanted gold seeds have been used to gate radiation delivery but the markers are invasive and gating significantly increases treatment time. Pretreatment acquisition of a respiratory correlated 4DCT allows for determination of accurate motion tracking which is useful in treatment planning. We design a patient-specific motion subspace and a deep convolutional neural network to recover anatomical positions from a single fluoroscopic projection in real-time. We use this deep network to appr…
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（香港中文大学（深圳）与中国科学院深圳先进技术研究院联合机构））；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
This paper presents a new way to study registration based trackers by decomposing them into three constituent sub modules: appearance model, state space model and search method. It is often the case that when a new tracker is introduced in literature, it only contributes to one or two of these sub modules while using existing methods for the rest. Since these are often selected arbitrarily by the authors, they may not be optimal for the new method. In such cases, our breakdown can help to experimentally find the best combination of methods for these sub modules while also providing a framework…
SLAM assisted 3D tracking system for laparoscopic surgery
United Imaging Research Institute of Intelligent Imaging（联影智能影像研究院）；University of Michigan（密歇根大学）；United Imaging（联影医疗）；United Imaging of Surgery（联影外科研究院）
GLIDE-Reg: Global-to-Local Deformable Registration Using Co-Optimized Foundation and Handcrafted Features
1 Medical \& Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, USA 2 Magnetic Resonance Research Labs, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, USA 3 Department of Radiation Oncology, David Geffen School of Medicine at UCLA, Los Angeles, USA
Tracking the Flight: Exploring a Computational Framework for Analyzing Escape Responses in Plains Zebra (Equus quagga)
Princeton University（普林斯顿大学）；Sainsbury Wellcome Centre, University College London（萨金斯伯里·韦科中心，伦敦大学学院）；Gatsby Computational Neuroscience Unit, University College London（加布里埃尔·加比计算神经科学单位，伦敦大学学院）
DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapy
Department of Computer Science and Software Engineering, Concordia University（计算机科学与软件工程系，康科迪亚大学）；Department of Electrical and Computer Engineering, Concordia University（电气与计算机工程系，康科迪亚大学）；Elekta Ltd.（Elekta公司）；Diagnos Medical Systems（Diagnos医疗系统）
This paper presents two visual trackers from the different paradigms of learning and registration based tracking and evaluates their application in image based visual servoing. They can track object motion with four degrees of freedom (DoF) which, as we will show here, is sufficient for many fine manipulation tasks. One of these trackers is a newly developed learning based tracker that relies on learning discriminative correlation filters while the other is a refinement of a recent 8 DoF RANSAC based tracker adapted with a new appearance model for tracking 4 DoF motion. Both trackers are shown…
K L University（K L大学）；MVSR Engineering College（MVSR工程学院）
Image registration is a classic problem of computer vision with several applications across areas like defence, remote sensing, medicine etc. Feature based image registration methods traditionally used hand-crafted feature extraction algorithms, which detect key points in an image and describe them using a region around the point. Such features are matched using a threshold either on distances or ratio of distances computed between the feature descriptors. Evolution of deep learning, in particular convolution neural networks, has enabled researchers to address several problems of vision such a…
Princeton University（普林斯顿大学）；Lewis-Sigler Institute for Integrative Genomics, Princeton University（普林斯顿大学刘易斯-西格勒综合基因组学研究所）；Princeton Neuroscience Institute, Princeton University（普林斯顿大学普林斯顿神经科学研究所）；Tufts University School of Medicine（塔夫茨大学医学院）
Advances in optical neuroimaging techniques now allow neural activity to be recorded with cellular resolution in awake and behaving animals. Brain motion in these recordings pose a unique challenge. The location of individual neurons must be tracked in 3D over time to accurately extract single neuron activity traces. Recordings from small invertebrates like C. elegans are especially challenging because they undergo very large brain motion and deformation during animal movement. Here we present an automated computer vision pipeline to reliably track populations of neurons with single neuron res…
Exploiting Longitudinal Context in Clinician-Verified Interactive Lesion Tracking
German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Germany（德国癌症研究中心（DKFZ）海德堡，医学图像计算部，德国）；Faculty of Mathematics and Computer Science, Heidelberg University, Germany（海德堡大学数学与计算机科学学院，德国）；HIDSS4Health -- Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany（HIDSS4Health——海德堡信息与数据科学健康学校，卡尔斯鲁厄/海德堡，德国）；Medical Faculty, Heidelberg University, Germany（海德堡大学医学学院，德国）；University Hospital Brandenburg an der Havel, Brandenburg Medical School Theodor Fontane, Germany（勃兰登堡运河大学医院，布兰登堡泰奥多尔·冯·_fontane医学学校，德国）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Germany（放射肿瘤科模式分析与学习组，海德堡大学医院，德国）
Geometry OR Tracker: Universal Geometric Operating Room Tracking
Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）；Hong Kong Institute of Science and Innovation Chinese Academy of Sciences（香港科学与创新研究院中国科学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation Chinese Academy of Sciences (CASIA)（自动化研究所中国科学院（CASIA））；Peking University（北京大学）；University of Trento, Trento, Italy（意大利特伦特大学）；Alibaba DAMO Academy（阿里达摩院）；Xiaomi Inc.（小米公司）；Technische Universität München（慕尼黑技术大学）
Geometry OR Tracker通过多视角几何校正和鲁棒3D点跟踪提升手术室多视角3D跟踪的几何一致性与准确性。
