Semantic segmentation of building facade is significant in various applications, such as urban building reconstruction and damage assessment. As there is a lack of 3D point clouds datasets related to the fine-grained building facade, we construct the first large-scale building facade point clouds benchmark dataset for semantic segmentation. The existing methods of semantic segmentation cannot fully mine the local neighborhood information of point clouds. Addressing this problem, we propose a learnable attention module that learns Dual Local Attention features, called DLA in this paper. The proposed DLA module consists of two blocks, including the self-attention block and attentive pooling block, which both embed an enhanced position encoding block. The DLA module could be easily embedded into various network architectures for point cloud segmentation, naturally resulting in a new 3D semantic segmentation network with an encoder-decoder architecture, called DLA-Net in this work. Extensive experimental results on our constructed building facade dataset demonstrate that the proposed DLA-Net achieves better performance than the state-of-the-art methods for semantic segmentation.
Urban facade segmentation from automatically acquired imagery, in contrast to traditional image segmentation, poses several unique challenges. 360-degree photospheres captured from vehicles are an effective way to capture a large number of images, but this data presents difficult-to-model warping and stitching artifacts. In addition, each pixel can belong to multiple facade elements, and different facade elements (e.g., window, balcony, sill, etc.) are correlated and vary wildly in their characteristics. In this paper, we propose three network architectures of varying complexity to achieve mul…
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Effectively parsing the facade is essential to 3D building reconstruction, which is an important computer vision problem with a large amount of applications in high precision map for navigation, computer aided design, and city generation for digital entertainments. To this end, the key is how to obtain the shape grammars from 2D images accurately and efficiently. Although enjoying the merits of promising results on the semantic parsing, deep learning methods cannot directly make use of the architectural rules, which play an important role for man-made structures. In this paper, we present a no…
BFA-YOLO: A balanced multiscale object detection network for building façade attachments detection
School of Geosciences, Yangtze University（长江大学地球科学学院）；State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Institute of Geospatial Information, Information Engineering University（信息工程大学地理空间信息学院）
Technical University of Munich (TUM)（慕尼黑工业大学）；University of Applied Science (HM)（应用科学大学（HM））；Bruno Kessler Foundation (FBK)（布鲁诺·凯斯勒基金会（FBK））
Semantic 3D building models are widely available and used in numerous applications. Such 3D building models display rich semantics but no façade openings, chiefly owing to their aerial acquisition techniques. Hence, refining models' façades using dense, street-level, terrestrial point clouds seems a promising strategy. In this paper, we propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. Voxels are also used to iden…
The parsing of windows in building facades is a long-desired but challenging task in computer vision. It is crucial to urban analysis, semantic reconstruction, lifecycle analysis, digital twins, and scene parsing amongst other building-related tasks that require high-quality semantic data. This article investigates the usage of the mask R-CNN framework to be used for window detection of facade imagery input. We utilize transfer learning to train our proposed method on COCO weights with our own collected dataset of street view images of facades to produce instance segmentations of our new windo…
Strengthening Layer Interaction via Dynamic Layer Attention
College of Computer Science, Sichuan University（四川大学计算机学院）；Clinical Medical College and The First Affiliated Hospital of Chengdu Medical College（成都医学院临床医学院及第一附属医院）
Dresden University of Technology（德累斯顿工业大学）；Zhejiang University（浙江大学）；Changzhou Microintelligence Co., Ltd.（常州微智有限公司）
Semantic segmentation profits from deep learning and has shown its possibilities in handling the graphical data from the on-site inspection. As a result, visual damage in the facade images should be detected. Attention mechanism and generative adversarial networks are two of the most popular strategies to improve the quality of semantic segmentation. With specific focuses on these two strategies, this paper adopts U-net, a representative convolutional neural network, as the primary network and presents a comparative study in two steps. First, cell images are utilized to respectively determine …
National Tsing Hua University（国立清华大学）；ETH Zürich（苏黎世联邦理工学院）；Meta Reality Labs Zürich（Meta苏黎世现实实验室）
Aggregating information from features across different layers is an essential operation for dense prediction models. Despite its limited expressiveness, feature concatenation dominates the choice of aggregation operations. In this paper, we introduce Attentive Feature Aggregation (AFA) to fuse different network layers with more expressive non-linear operations. AFA exploits both spatial and channel attention to compute weighted average of the layer activations. Inspired by neural volume rendering, we extend AFA with Scale-Space Rendering (SSR) to perform late fusion of multi-scale predictions.…
Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
The ability to recognize the position and order of the floor-level lines that divide adjacent building floors can benefit many applications, for example, urban augmented reality (AR). This work tackles the problem of locating floor-level lines in street-view images, using a supervised deep learning approach. Unfortunately, very little data is available for training such a network $-$ current street-view datasets contain either semantic annotations that lack geometric attributes, or rectified facades without perspective priors. To address this issue, we first compile a new dataset and develop a…
Guangzhou Institute of Technology, Xidian University（西安电子科技大学广州研究院）；Northwestern Polytechnical University（西北工业大学）
Existing methods for anomaly detection based on memory-augmented autoencoder (AE) have the following drawbacks: (1) Establishing a memory bank requires additional memory space. (2) The fixed number of prototypes from subjective assumptions ignores the data feature differences and diversity. To overcome these drawbacks, we introduce DLAN-AC, a Dynamic Local Aggregation Network with Adaptive Clusterer, for anomaly detection. First, The proposed DLAN can automatically learn and aggregate high-level features from the AE to obtain more representative prototypes, while freeing up extra memory space.…
Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray casting and Bayesian networks
Technical University of Munich（慕尼黑工业大学）；University of Oxford（牛津大学）；Bruno Kessler Foundation（布鲁诺·凯斯勒基金会）；Munich University of Applied Sciences（慕尼黑应用技术大学）
Representing features at multiple scales is of great importance for numerous vision tasks. Recent advances in backbone convolutional neural networks (CNNs) continually demonstrate stronger multi-scale representation ability, leading to consistent performance gains on a wide range of applications. However, most existing methods represent the multi-scale features in a layer-wise manner. In this paper, we propose a novel building block for CNNs, namely Res2Net, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a…
National Open University of Nigeria（尼日利亚国立开放大学）；Sure-GP Ltd（Sure-GP有限公司）
With the rise in militant activity and rogue behaviour in oil and gas regions around the world, oil pipeline disturbances is on the increase leading to huge losses to multinational operators and the countries where such facilities exist. However, this situation can be averted if adequate predictive monitoring schemes are put in place. We propose in the first part of this paper, an artificial intelligence predictive monitoring system capable of predictive classification and pattern recognition of pipeline datasets. The predictive system is based on a highly sparse predictive Deviant Learning Al…
East China Normal University（华东师范大学）；Videt Tech Ltd.（维迪特科技有限公司）
The document layout analysis (DLA) aims to split the document image into different interest regions and understand the role of each region, which has wide application such as optical character recognition (OCR) systems and document retrieval. However, it is a challenge to build a DLA system because the training data is very limited and lacks an efficient model. In this paper, we propose an end-to-end united network named Dynamic Residual Fusion Network (DRFN) for the DLA task. Specifically, we design a dynamic residual feature fusion module which can fully utilize low-dimensional information a…
BCE-Net: Reliable Building Footprints Change Extraction based on Historical Map and Up-to-Date Images using Contrastive Learning
Southwest Jiaotong University（西南交通大学）；Central South University（中南大学）；Anhui University of Science and Technology（安徽理工大学）；Guizhou Polytechnic of Construction（贵州建设职业技术学院）
Recent developments in the field of deep learning for 3D data have demonstrated promising potential for end-to-end learning directly from point clouds. However, many real-world point clouds contain a large class im-balance due to the natural class im-balance observed in nature. For example, a 3D scan of an urban environment will consist mostly of road and facade, whereas other objects such as poles will be under-represented. In this paper we address this issue by employing a weighted augmentation to increase classes that contain fewer points. By mitigating the class im-balance present in the d…
Reconstructing Building Height from Spaceborne TomoSAR Point Clouds Using a Dual-Topology Network
The Chair of Data Science in Earth Observation, Technical University of Munich (TUM)（地球观测数据科学教授职位，慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；School of Engineering and Design, Technical University of Munich (TUM)（工程与设计学院，慕尼黑技术大学）
In this work, we propose a novel neural network focusing on semantic labeling of ALS point clouds, which investigates the importance of long-range spatial and channel-wise relations and is termed as global relation-aware attentional network (GraNet). GraNet first learns local geometric description and local dependencies using a local spatial discrepancy attention convolution module (LoSDA). In LoSDA, the orientation information, spatial distribution, and elevation differences are fully considered by stacking several local spatial geometric learning modules and the local dependencies are embedd…
Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences（中国科学院遥感与数字地球研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）
Point cloud classification plays an important role in a wide range of airborne light detection and ranging (LiDAR) applications, such as topographic mapping, forest monitoring, power line detection, and road detection. However, due to the sensor noise, high redundancy, incompleteness, and complexity of airborne LiDAR systems, point cloud classification is challenging. In this paper, we proposed a directionally constrained fully convolutional neural network (D-FCN) that can take the original 3D coordinates and LiDAR intensity as input; thus, it can directly apply to unstructured 3D point clouds…
Deep Learning based Joint Geometry and Attribute Up-sampling for Large-Scale Colored Point Clouds
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen Campus（中山大学电子与通信工程学院，深圳校区）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（中国电子科技大学深圳研究院）；Department of Computing and Decision Sciences, Lingnan University（岭南大学计算机与决策科学系）
