This paper presents an efficient approach for solving jointly facade registration and semantic segmentation. Progress in facade detection and recognition enable good initialization for the registration of a reference facade to a newly acquired target image. We propose here to rely on semantic segmentation to improve the accuracy of that initial registration. Simultaneously we aim to improve the quality of the semantic segmentation through the registration. These two problems are jointly solved in a Expectation-Maximization framework. We especially introduce a bayesian model that use prior semantic segmentation as well as geometric structure of the facade reference modeled by $L_p$ Gaussian Mixtures. We show the advantages of our method in term of robustness to clutter and change of illumination on urban images from various database.
Urban facade segmentation from automatically acquired imagery, in contrast to traditional image segmentation, poses several unique challenges. 360-degree photospheres captured from vehicles are an effective way to capture a large number of images, but this data presents difficult-to-model warping and stitching artifacts. In addition, each pixel can belong to multiple facade elements, and different facade elements (e.g., window, balcony, sill, etc.) are correlated and vary wildly in their characteristics. In this paper, we propose three network architectures of varying complexity to achieve mul…
Semantic segmentation of building facade is significant in various applications, such as urban building reconstruction and damage assessment. As there is a lack of 3D point clouds datasets related to the fine-grained building facade, we construct the first large-scale building facade point clouds benchmark dataset for semantic segmentation. The existing methods of semantic segmentation cannot fully mine the local neighborhood information of point clouds. Addressing this problem, we propose a learnable attention module that learns Dual Local Attention features, called DLA in this paper. The pro…
The parsing of windows in building facades is a long-desired but challenging task in computer vision. It is crucial to urban analysis, semantic reconstruction, lifecycle analysis, digital twins, and scene parsing amongst other building-related tasks that require high-quality semantic data. This article investigates the usage of the mask R-CNN framework to be used for window detection of facade imagery input. We utilize transfer learning to train our proposed method on COCO weights with our own collected dataset of street view images of facades to produce instance segmentations of our new windo…
National University of Ireland Maynooth（爱尔兰国立梅努斯大学）；Université Paris-Est（巴黎东大学）；Institut Géographique National（法国国家地理研究院）；University of the Basque Country（巴斯克大学）
In this paper we present a practical approach for generating an occlusion-free textured 3D map of urban facades by the synergistic use of terrestrial images, 3D point clouds and area-based information. Particularly in dense urban environments, the high presence of urban objects in front of the facades causes significant difficulties for several stages in computational building modeling. Major challenges lie on the one hand in extracting complete 3D facade quadrilateral delimitations and on the other hand in generating occlusion-free facade textures. For these reasons, we describe a straightfor…
Technical University of Munich (TUM)（慕尼黑工业大学）；University of Applied Science (HM)（应用科学大学（HM））；Bruno Kessler Foundation (FBK)（布鲁诺·凯斯勒基金会（FBK））
Semantic 3D building models are widely available and used in numerous applications. Such 3D building models display rich semantics but no façade openings, chiefly owing to their aerial acquisition techniques. Hence, refining models' façades using dense, street-level, terrestrial point clouds seems a promising strategy. In this paper, we propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. Voxels are also used to iden…
Satellite imagery analytics have numerous human development and disaster response applications, particularly when time series methods are involved. For example, quantifying population statistics is fundamental to 67 of the 231 United Nations Sustainable Development Goals Indicators, but the World Bank estimates that over 100 countries currently lack effective Civil Registration systems. To help address this deficit and develop novel computer vision methods for time series data, we present the Multi-Temporal Urban Development SpaceNet (MUDS, also known as SpaceNet 7) dataset. This open source d…
Research Center for Advanced Science and Technology, the University of Tokyo（东京大学先进科学技术研究中心）；the University of Tokyo（东京大学）；Softbank Corp.（软银公司）
The disorder of urban streetscapes would negatively affect people's perception of their aesthetic quality. The presence of billboards on building facades has been regarded as an important factor of the disorder, but its quantification methodology has not yet been developed in a scalable manner. To fill the gap, this paper reports the performance of our deep learning model on a unique data set prepared in Tokyo to recognize the areas covered by facades and billboards in streetscapes, respectively. The model achieved 63.17 % of accuracy, measured by Intersection-over-Union (IoU), thus enabling r…
The motivation of this paper is to address the problem of registering airborne LiDAR data and optical aerial or satellite imagery acquired from different platforms, at different times, with different points of view and levels of detail. In this paper, we present a robust registration method based on building regions, which are extracted from optical images using mean shift segmentation, and from LiDAR data using a 3D point cloud filtering process. The matching of the extracted building segments is then carried out using Graph Transformation Matching (GTM) which allows to determine a common pat…
In this paper, based on the assumption that the object boundaries (e.g., buildings) from the over-view data should coincide with footprints of façade 3D points generated from street-view photogrammetric images, we aim to address this problem by proposing a fully automated geo-registration method for cross-view data, which utilizes semantically segmented object boundaries as view-invariant features under a global optimization framework through graph-matching: taking the over-view point clouds generated from stereo/multi-stereo satellite images and the street-view point clouds generated from mon…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；NAVER LABS（NAVER实验室）
Global registration using 3D point clouds is a crucial technology for mobile platforms to achieve localization or manage loop-closing situations. In recent years, numerous researchers have proposed global registration methods to address a large number of outlier correspondences. Unfortunately, the degeneracy problem, which represents the phenomenon in which the number of estimated inliers becomes lower than three, is still potentially inevitable. To tackle the problem, a degeneracy-robust decoupling-based global registration method is proposed, called Quatro. In particular, our method employs …
Télécom ParisTech, Université Paris-Saclay（巴黎萨克雷大学电信学院）；Univ Lyon, UJM-Saint-Etienne（里昂大学，圣艾蒂安大学）；ONERA, The French Aerospace Laboratory（法国航空航天研究院）；Institute of Electronics and Telecommunications of Rennes (IETR), University of Rennes 1（雷恩第一大学雷恩电子与电信研究所）
SAR (Synthetic Aperture Radar) tomography reconstructs 3-D volumes from stacks of SAR images. High-resolution satellites such as TerraSAR-X provide images that can be combined to produce 3-D models. In urban areas, sparsity priors are generally enforced during the tomographic inversion process in order to retrieve the location of scatterers seen within a given radar resolution cell. However, such priors often miss parts of the urban surfaces. Those missing parts are typically regions of flat areas such as ground or rooftops. This paper introduces a surface segmentation algorithm based on the c…
Deep Learning-based Scalable Image-to-3D Facade Parser for Generating Thermal 3D Building Models
Department of Computer Science and Engineering, Chalmers University of Technology（计算机科学与工程系，查尔姆斯理工大学）；Department of Architecture and Civil Engineering, Chalmers University of Technology（建筑与土木工程系，查尔姆斯理工大学）
The problem visual place recognition is commonly used strategy for localization. Most successful appearance based methods typically rely on a large database of views endowed with local or global image descriptors and strive to retrieve the views of the same location. The quality of the results is often affected by the density of the reference views and the robustness of the image representation with respect to viewpoint variations, clutter and seasonal changes. In this work we present an approach for geo-locating a novel view and determining camera location and orientation using a map and a sp…
Applications based on synergistic integration of optical imagery and LiDAR data are receiving a growing interest from the remote sensing community. However, a misaligned integration between these datasets may fail to fully profit the potential of both sensors. In this regard, an optimum fusion of optical imagery and LiDAR data requires an accurate registration. This is a complex problem since a versatile solution is still missing, especially when considering the context where data are collected at different times, from different platforms, under different acquisition configurations. This paper…
BCE-Net: Reliable Building Footprints Change Extraction based on Historical Map and Up-to-Date Images using Contrastive Learning
Southwest Jiaotong University（西南交通大学）；Central South University（中南大学）；Anhui University of Science and Technology（安徽理工大学）；Guizhou Polytechnic of Construction（贵州建设职业技术学院）
Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray casting and Bayesian networks
Technical University of Munich（慕尼黑工业大学）；University of Oxford（牛津大学）；Bruno Kessler Foundation（布鲁诺·凯斯勒基金会）；Munich University of Applied Sciences（慕尼黑应用技术大学）
GS4City: Hierarchical Semantic Gaussian Splatting via City-Model Priors
Technical University of Munich (TUM)（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；CV4DT, University of Cambridge（剑桥大学CV4DT）
Poles and building edges are frequently observable objects on urban roads, conveying reliable hints for various computer vision tasks. To repetitively extract them as features and perform association between discrete LiDAR frames for registration, we propose the first learning-based feature segmentation and description model for 3D lines in LiDAR point cloud. To train our model without the time consuming and tedious data labeling process, we first generate synthetic primitives for the basic appearance of target lines, and build an iterative line auto-labeling process to gradually refine line l…
Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Review
University of Liverpool（利物浦大学）；Beihang University（北京航空航天大学）；Cambridge University（剑桥大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Hong Kong Polytechnic University（香港理工大学）；University of California, Berkeley（加州大学伯克利分校）
Mapping and localization is a critical module of autonomous driving, and significant achievements have been reached in this field. Beyond Global Navigation Satellite System (GNSS), research in point cloud registration, visual feature matching, and inertia navigation has greatly enhanced the accuracy and robustness of mapping and localization in different scenarios. However, highly urbanized scenes are still challenging: LIDAR- and camera-based methods perform poorly with numerous dynamic objects; the GNSS-based solutions experience signal loss and multipath problems; the inertia measurement un…
InfraRed Investigation in Singapore (IRIS) Observatory: Urban heat island contributors and mitigators analysis using neighborhood-scale thermal imaging
Berkeley Education Alliance for Research in Singapore（新加坡伯克利教育研究联盟）；College of Design and Engineering, National University of Singapore (NUS)（新加坡国立大学设计与工程学院）
L2M-Reg: Building-level Uncertainty-aware Registration of Outdoor LiDAR Point Clouds and Semantic 3D City Models
Chair of Engineering Geodesy, TUM School of Engineering and Design, Technical University of Munich（工程测量学教授会，技术大学慕尼黑工程与设计学院）；Chair of Geoinformatics, TUM School of Engineering and Design, Technical University of Munich（地理信息学教授会，技术大学慕尼黑工程与设计学院）
Geo-Registration of Terrestrial LiDAR Point Clouds with Satellite Images without GNSS
Department of Computer Science and Software Engineering, The University of Western Australia（计算机科学与软件工程系，西澳大学）；School of Engineering and Technology, University of New South Wales（工程与技术学院，新南威尔士大学）
OpenFACADES: An Open Framework for Architectural Caption and Attribute Data Enrichment via Street View Imagery
Department of Architecture, National University of Singapore（建筑系，新加坡国立大学）；Department of Electrical and Computer Engineering, National University of Singapore（电气与计算机工程系，新加坡国立大学）；School of Artificial Intelligence, Shenzhen Technology University（人工智能学院，深圳科技大学）；Department of Real Estate, National University of Singapore（房地产系，新加坡国立大学）
Regist3R: Incremental Registration with Stereo Foundation Model
College of Computer Science and Technology（计算机科学与技术学院）；National Key Laboratory of Parallel and Distributed Computing（并行与分布式计算国家重点实验室）；National University of Defense Technology（国防科技大学）
Analyzing the impact of semantic LoD3 building models on image-based vehicle localization
Technical University of Munich（慕尼黑工业大学）；TUM School of Engineering and Design（慕尼黑工业大学工程与设计学院）；D Mapping Solutions GmbH（3D测绘解决方案有限公司）；Hochschule München University of Applied Sciences（慕尼黑应用科学大学）
SemCityLoc: Aerial 6DoF Localization Using Semantic 3D City Models
Technical University of Munich（慕尼黑工业大学）；University of Cambridge（剑桥大学）；University of Science and Technology of China（中国科学技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；ETH Zurich（苏黎世联邦理工学院）
Spectral methods such as the improved Fourier Mellin Invariant (iFMI) transform have proved faster, more robust and accurate than feature based methods on image registration. However, iFMI is restricted to work only when the camera moves in 2D space and has not been applied on omni-cameras images so far. In this work, we extend the iFMI method and apply a motion model to estimate an omni-camera's pose when it moves in 3D space. This is particularly useful in field robotics applications to get a rapid and comprehensive view of unstructured environments, and to estimate robustly the robot pose. …
Elevation Estimation-Driven Building 3D Reconstruction from Single-View Remote Sensing Imagery
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Geovis Technology Co., Ltd.（中科星图股份有限公司）
To Glue or Not to Glue? Classical vs Learned Image Matching for Mobile Mapping Cameras to Textured Semantic 3D Building Models
Professorship of Photogrammetry and Remote Sensing, TUM School of Engineering and Design, Technical University of Munich（摄影测量与遥感教授职位，技术大学慕尼黑工程与设计学院，技术大学慕尼黑）
Autonomous Systems Lab, ETH Zurich（苏黎世联邦理工学院自主系统实验室）
Global registration of multi-view robot data is a challenging task. Appearance-based global localization approaches often fail under drastic view-point changes, as representations have limited view-point invariance. This work is based on the idea that human-made environments contain rich semantics which can be used to disambiguate global localization. Here, we present X-View, a Multi-View Semantic Global Localization system. X-View leverages semantic graph descriptor matching for global localization, enabling localization under drastically different view-points. While the approach is general i…
