A semantic feature extraction method for multitemporal high resolution aerial image registration is proposed in this paper. These features encode properties or information about temporally invariant objects such as roads and help deal with issues such as changing foliage in image registration, which classical handcrafted features are unable to address. These features are extracted from a semantic segmentation network and have shown good robustness and accuracy in registering aerial images across years and seasons in the experiments.
This paper addresses the task of time separated aerial image registration. The ability to solve this problem accurately and reliably is important for a variety of subsequent image understanding applications. The principal challenge lies in the extent and nature of transient appearance variation that a land area can undergo, such as that caused by the change in illumination conditions, seasonal variations, or the occlusion by non-persistent objects (people, cars). Our work introduces several novelties: (i) unlike all previous work on aerial image registration, we approach the problem using a se…
ALTEN Company（阿尔滕公司）；stimulating innovation center of ALTEN Company（阿尔滕公司创新中心）；Tyrix Aerospace Company（特里克斯航空航天公司）
This paper proposes to study an alternative technology to the GPS system on fixed wing aircraft using the aerial shots of landscapes from a ventral monocular camera integrated into the aircraft and based on the technology of image registration for aircraft geolocation purpose. Different types of use of the image registration technology exist: the relative registration and the absolute registration. The relative one is able to readjust position of the aircraft from two successive aerial shots by knowing the aircraft s position of image 1 and the overlap between the two images. The absolute regi…
Aerial image registration or matching is a geometric process of aligning two aerial images captured in different environments. Estimating the precise transformation parameters is hindered by various environments such as time, weather, and viewpoints. The characteristics of the aerial images are mainly composed of a straight line owing to building and road. Therefore, the straight lines are distorted when estimating homography parameters directly between two images. In this paper, we propose a deep homography alignment network to precisely match two aerial images by progressively estimating the…
Southwest Jiaotong University（西南交通大学）；Xinyang Normal University（信阳师范大学）
Over the past few decades, with the rapid development of global aerospace and aerial remote sensing technology, the types of sensors have evolved from the traditional monomodal sensors (e.g., optical sensors) to the new generation of multimodal sensors [e.g., multispectral, hyperspectral, light detection and ranging (LiDAR) and synthetic aperture radar (SAR) sensors]. These advanced devices can dynamically provide various and abundant multimodal remote sensing images with different spatial, temporal, and spectral resolutions according to different application requirements. Since then, it is of…
Institute for Applied Life Sciences, University of Massachusetts（马萨诸塞大学应用生命科学研究所）；University of Massachusetts（马萨诸塞大学）
Advancement of imaging techniques enables consecutive image sequences to be acquired for quality monitoring of manufacturing production lines. Registration for these image sequences is essential for in-line pattern inspection and metrology, e.g., in the printing process of flexible electronics. However, conventional image registration algorithms cannot produce accurate results when the images contain many similar and deformable patterns in the manufacturing process. Such a failure originates from a fact that the conventional algorithms only use the spatial and pixel intensity information for r…
Applications based on synergistic integration of optical imagery and LiDAR data are receiving a growing interest from the remote sensing community. However, a misaligned integration between these datasets may fail to fully profit the potential of both sensors. In this regard, an optimum fusion of optical imagery and LiDAR data requires an accurate registration. This is a complex problem since a versatile solution is still missing, especially when considering the context where data are collected at different times, from different platforms, under different acquisition configurations. This paper…
We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space - the motion matrix - which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation. More precisely, the motion matrix allows to transport the recovered motion from one subject to another simulating for example a pathological motion in a healthy subject without the need for inter-subject registration. The method is b…
University College London（伦敦大学学院）；InstaDeep；University College London Hospital NHS Foundation Trust（伦敦大学学院医院NHS基金会信托）；University of Oxford（牛津大学）
Morphological analysis of longitudinal MR images plays a key role in monitoring disease progression for prostate cancer patients, who are placed under an active surveillance program. In this paper, we describe a learning-based image registration algorithm to quantify changes on regions of interest between a pair of images from the same patient, acquired at two different time points. Combining intensity-based similarity and gland segmentation as weak supervision, the population-data-trained registration networks significantly lowered the target registration errors (TREs) on holdout patient data…
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）
Multilevel strategies are an integral part of many image registration algorithms. These strategies are very well-known for avoiding undesirable local minima, providing an outstanding initial guess, and reducing overall computation time. State-of-the-art multilevel strategies build a hierarchy of discretization in the spatial dimensions. In this paper, we present a spatio-temporal strategy, where we introduce a hierarchical discretization in the temporal dimension at each spatial level. This strategy is suitable for a motion estimation problem where the motion is assumed smooth over time. Our s…
The Chinese University of Hong Kong（香港中文大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Tencent Jarvis Lab（腾讯医疗健康（深圳）有限公司（腾讯医疗AI实验室/腾讯天衍实验室））；Tsinghua University（清华大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
In order to tackle the difficulty associated with the ill-posed nature of the image registration problem, regularization is often used to constrain the solution space. For most learning-based registration approaches, the regularization usually has a fixed weight and only constrains the spatial transformation. Such convention has two limitations: (i) Besides the laborious grid search for the optimal fixed weight, the regularization strength of a specific image pair should be associated with the content of the images, thus the "one value fits all" training scheme is not ideal; (ii) Only spatiall…
TLRN: Temporal Latent Residual Networks For Large Deformation Image Registration
Department of Electrical and Computer Engineering, University of Virginia, USA（电气与计算机工程系，弗吉尼亚大学）；Department of Computer Science, University of Virginia, USA（计算机科学系，弗吉尼亚大学）
We introduce a region-specific diffeomorphic metric mapping (RDMM) registration approach. RDMM is non-parametric, estimating spatio-temporal velocity fields which parameterize the sought-for spatial transformation. Regularization of these velocity fields is necessary. However, while existing non-parametric registration approaches, e.g., the large displacement diffeomorphic metric mapping (LDDMM) model, use a fixed spatially-invariant regularization our model advects a spatially-varying regularizer with the estimated velocity field, thereby naturally attaching a spatio-temporal regularizer to d…
State Key Laboratory of Information Engineering in Surveying Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
The emerging drone aerial survey has the advantages of low cost, high efficiency, and flexible use. However, UAVs are often equipped with cheap POS systems and non-measurement cameras, and their flight attitudes are easily affected. How to realize the large-scale mapping of UAV image-free control supported by POS faces many technical problems. The most basic and important core technology is how to accurately realize the absolute orientation of images through advanced aerial triangulation technology. In traditional aerial triangulation, image matching algorithms are constrained to varying degre…
Co-Registration of aerial imagery and Light Detection and Ranging (LiDAR) data is quilt challenging because the different imaging mechanism causes significant geometric and radiometric distortions between such data. To tackle the problem, this paper proposes an automatic registration method based on structural features and three-dimension (3D) phase correlation. In the proposed method, the LiDAR point cloud data is first transformed into the intensity map, which is used as the reference image. Then, we employ the Fast operator to extract uniformly distributed interest points in the aerial imag…
This letter proposes a synthetic aperture radar (SAR) image registration method named Feature-Area Optimization (FAO). First, the traditional area-based optimization model is reconstructed and decomposed into three key but uncertain factors: initialization, slice set and regularization. Next, structural features are extracted by scale invariant feature transform (SIFT) in dual-resolution space (SIFT-DRS), a novel SIFT-Like method dedicated to FAO. Then, the three key factors are determined based on these features. Finally, solving the factor-determined optimization model can get the registrati…
Continuous sPatial-Temporal Deformable Image Registration (CPT-DIR) for motion modelling in radiotherapy: beyond classic voxel-based methods
Center for Proton Therapy, Paul Scherrer Institut（保罗·施特格研究所质子治疗中心）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Department of Physics, ETH Zürich（苏黎世联邦理工学院物理系）；Department of Automation, Tsinghua Univeristy（清华大学自动化系）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
École Polytechnique de Montréal（蒙特利尔高等理工学院）；École de technologie supérieure（高等工程技术学院）
In this paper, an automatic method is proposed to perform image registration in visible and infrared pair of video sequences for multiple targets. In multimodal image analysis like image fusion systems, color and IR sensors are placed close to each other and capture a same scene simultaneously, but the videos are not properly aligned by default because of different fields of view, image capturing information, working principle and other camera specifications. Because the scenes are usually not planar, alignment needs to be performed continuously by extracting relevant common information. In th…
The implementation of automatic image registration is still difficult in various applications. In this paper, an automatic image registration approach through line-support region segmentation and geometrical outlier removal (ALRS-GOR) is proposed. This new approach is designed to address the problems associated with the registration of images with affine deformations and inconsistent content, such as remote sensing images with different spectral content or noise interference, or map images with inconsistent annotations. To begin with, line-support regions, namely a straight region whose points…
Fraunhofer IOSB（弗劳恩霍夫IOSB研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
The comparison of current image data with existing 3D model data of a scene provides an efficient method to keep models up to date. In order to transfer information between 2D and 3D data, a preliminary co-registration is necessary. In this paper, we present a concept to automatically co-register aerial imagery and untextured 3D model data. To refine a given initial camera pose, our algorithm computes dense correspondence fields using SIFT flow between gradient representations of the model and camera image, from which 2D-3D correspondences are obtained. These correspondences are then used in a…
In this paper, we address the registration of historical WWII images to present-day ortho-photo maps for the purpose of geolocalization. Due to the challenging nature of this problem, we propose to register the images jointly as a group rather than in a step-by-step manner. To this end, we exploit Hough Voting spaces as pairwise registration estimators and show how they can be integrated into a probabilistic groupwise registration framework that can be efficiently optimized. The feature-based nature of our registration framework allows to register images with a-priori unknown translational and…
University College London（伦敦大学学院）；UCL Institute for Liver and Digestive Health（伦敦大学学院肝脏与消化健康研究所）
Purpose: Probe-based Confocal Laser Endomicroscopy (pCLE) is a recent imaging modality that allows performing in vivo optical biopsies. The design of pCLE hardware, and its reliance on an optical fibre bundle, fundamentally limits the image quality with a few tens of thousands fibres, each acting as the equivalent of a single-pixel detector, assembled into a single fibre bundle. Video-registration techniques can be used to estimate high-resolution (HR) images by exploiting the temporal information contained in a sequence of low-resolution (LR) images. However, the alignment of LR frames, requi…
University of Bologna（博洛尼亚大学）；University of Calabria（卡拉布里亚大学）
We present a new system based on tracking the temporal evolution of stroke lesions using an image registration technique on CT exams of the patient's brain. The system is able to compare past CT exams with the most recent one related to stroke event in order to evaluate past lesions which are not related to stroke. Then, it can compare recent CT exams related to the current stroke for assessing the evolution of the lesion over time. A new similarity measure is also introduced for the comparison of the source and target images during image registration. It will result in a cheaper, faster and m…
Eurecat Centre Tecnològic de Catalunya（加泰罗尼亚埃urecat技术中心）；Universitat Pompeu Fabra（庞培法布拉大学）；Vall d’Hebron University Hospital（瓦尔德希伯伦大学医院）；ICREA（加泰罗尼亚研究与高等研究院）
Lung cancer follow-up is a complex, error prone, and time consuming task for clinical radiologists. Several lung CT scan images taken at different time points of a given patient need to be individually inspected, looking for possible cancerogenous nodules. Radiologists mainly focus their attention in nodule size, density, and growth to assess the existence of malignancy. In this study, we present a novel method based on a 3D siamese neural network, for the re-identification of nodules in a pair of CT scans of the same patient without the need for image registration. The network was integrated …
We present a robust method to correct for motion and deformations for in-utero volumetric MRI time series. Spatio-temporal analysis of dynamic MRI requires robust alignment across time in the presence of substantial and unpredictable motion. We make a Markov assumption on the nature of deformations to take advantage of the temporal structure in the image data. Forward message passing in the corresponding hidden Markov model (HMM) yields an estimation algorithm that only has to account for relatively small motion between consecutive frames. We demonstrate the utility of the temporal model by sh…
Massachusetts Institute of Technology（麻省理工学院）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Washington University in St. Louis（圣路易斯华盛顿大学）；Shanghai Jiao Tong University（上海交通大学）
We present a robust method to correct for motion in volumetric in-utero MRI time series. Time-course analysis for in-utero volumetric MRI time series often suffers from substantial and unpredictable fetal motion. Registration provides voxel correspondences between images and is commonly employed for motion correction. Current registration methods often fail when aligning images that are substantially different from a template (reference image). To achieve accurate and robust alignment, we make a Markov assumption on the nature of motion and take advantage of the temporal smoothness in the imag…
E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning
Intel Labs（英特尔实验室）；Apple Inc.（苹果公司）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU（厦门大学多媒体可信感知与高效计算重点实验室）；Yancheng Institute Of Technology（盐城工学院）
Tongji University（同济大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Temporal contexts among consecutive frames are far from being fully utilized in existing visual trackers. In this work, we present TCTrack, a comprehensive framework to fully exploit temporal contexts for aerial tracking. The temporal contexts are incorporated at \textbf{two levels}: the extraction of \textbf{features} and the refinement of \textbf{similarity maps}. Specifically, for feature extraction, an online temporally adaptive convolution is proposed to enhance the spatial features using temporal information, which is achieved by dynamically calibrating the convolution weights according …
Wuhan University（武汉大学）；Technical University of Munich（慕尼黑工业大学）；German Aerospace Center（德国航空航天中心）
Unmanned aerial vehicles (UAVs) are now widely applied to data acquisition due to its low cost and fast mobility. With the increasing volume of aerial videos, the demand for automatically parsing these videos is surging. To achieve this, current researches mainly focus on extracting a holistic feature with convolutions along both spatial and temporal dimensions. However, these methods are limited by small temporal receptive fields and cannot adequately capture long-term temporal dependencies which are important for describing complicated dynamics. In this paper, we propose a novel deep neural …
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）
Multi-pedestrian tracking in aerial imagery has several applications such as large-scale event monitoring, disaster management, search-and-rescue missions, and as input into predictive crowd dynamic models. Due to the challenges such as the large number and the tiny size of the pedestrians (e.g., 4 x 4 pixels) with their similar appearances as well as different scales and atmospheric conditions of the images with their extremely low frame rates (e.g., 2 fps), current state-of-the-art algorithms including the deep learning-based ones are unable to perform well. In this paper, we propose AerialM…
New York University Abu Dhabi（纽约大学阿布扎比分校）；New York University（纽约大学）；NYU Tandon School of Engineering（纽约大学坦登工程学院）
Global registration of point clouds aims to find an optimal alignment of a sequence of 2D or 3D point sets. In this paper, we present a novel method that takes advantage of current deep learning techniques for unsupervised learning of global registration from a temporal sequence of point clouds. Our key novelty is that we introduce a deep Spatio-Temporal REPresentation (STREP) feature, which describes the geometric essence of both temporal and spatial relationship of the sequence of point clouds acquired with sensors in an unknown environment. In contrast to the previous practice that treats e…
Shantou University（汕头大学）；Sun Yat-sen University（中山大学）；First Affiliated Hospital of Shantou University（汕头大学第一附属医院）
Deformable image registration can obtain dynamic information about images, which is of great significance in medical image analysis. The unsupervised deep learning registration method can quickly achieve high registration accuracy without labels. However, these methods generally suffer from uncorrelated features, poor ability to register large deformations and details, and unnatural deformation fields. To address the issues above, we propose an unsupervised multi-scale correlation iterative registration network (SearchMorph). In the proposed network, we introduce a correlation layer to strengt…
MemReg: Streaming Outdoor LiDAR Point Cloud Registration with Hybrid Memory Buffers
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）；University of Twente（特文特大学）；University of Cambridge（剑桥大学）；University of Bath（巴斯大学）；Phigent Robotics（飞劲机器人公司）
We propose to learn a probabilistic motion model from a sequence of images. Besides spatio-temporal registration, our method offers to predict motion from a limited number of frames, useful for temporal super-resolution. The model is based on a probabilistic latent space and a novel temporal dropout training scheme. This enables simulation and interpolation of realistic motion patterns given only one or any subset of frames of a sequence. The encoded motion also allows to be transported from one subject to another without the need of inter-subject registration. An unsupervised generative defor…
THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage
University of Arkansas（亚拉巴马大学）；University of Science, VNU-HCM（科学大学，VNU-HCM）；Vietnam National University, Ho Chi Minh（越南国家大学，胡志明市）；Ohio State University（俄亥俄州立大学）
TempDiffReg: Temporal Diffusion Model for Non-Rigid 2D-3D Vascular Registration
School of Computer Science and Engineering, Beihang University, Beijing, China（北京航空航天大学计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（中国科学院深圳先进技术研究所）；School of Computer Science and Control Engineering, Shenzhen University of Advanced Technology, Shenzhen, China（深圳先进技术大学计算机科学与控制工程学院）；Department of Interventional Radiology, Shenzhen People’s Hospital, Shenzhen, China（深圳人民医院介入放射科）
Wuhan University（武汉大学）；Huazhong University of Science and Technology（华中科技大学）；University of Twente（特温特大学）；Chinese Academy of Sciences（中国科学院）；Cornell Tech and Cornell University（康奈尔科技学院和康奈尔大学）；University of Rochester（罗切斯特大学）；German Aerospace Center (DLR)（德国航空航天中心（DLR））；University POLITEHNICA of Bucharest（布加勒斯特理工大学）；Ca’ Foscari University of Venice（威尼斯卡福斯卡里大学）；Hikvision Research Institute（海康威视研究院）；Alibaba Group（阿里巴巴集团）；Xidian University（西安电子科技大学）
This report summarizes the results of Learning to Understand Aerial Images (LUAI) 2021 challenge held on ICCV 2021, which focuses on object detection and semantic segmentation in aerial images. Using DOTA-v2.0 and GID-15 datasets, this challenge proposes three tasks for oriented object detection, horizontal object detection, and semantic segmentation of common categories in aerial images. This challenge received a total of 146 registrations on the three tasks. Through the challenge, we hope to draw attention from a wide range of communities and call for more efforts on the problems of learning…
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）
Semantic segmentation of aerial videos has been extensively used for decision making in monitoring environmental changes, urban planning, and disaster management. The reliability of these decision support systems is dependent on the accuracy of the video semantic segmentation algorithms. The existing CNN based video semantic segmentation methods have enhanced the image semantic segmentation methods by incorporating an additional module such as LSTM or optical flow for computing temporal dynamics of the video which is a computational overhead. The proposed research work modifies the CNN archite…
A useful application of event sensing is visual odometry, especially in settings that require high-temporal resolution. The state-of-the-art method of contrast maximisation recovers the motion from a batch of events by maximising the contrast of the image of warped events. However, the cost scales with image resolution and the temporal resolution can be limited by the need for large batch sizes to yield sufficient structure in the contrast image. In this work, we propose spatiotemporal registration as a compelling technique for event-based rotational motion estimation. We theoretcally justify …
Echocardiography Segmentation Using Neural ODE-based Diffeomorphic Registration Field
University of Engineering and Technology, Vietnam National University（越南国立大学工程技术大学）；VinUniversity（文梅大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
