Road Segmentation for Remote Sensing Images using Adversarial Spatial Pyramid Networks
Shanghai Jiao Tong University（上海交通大学）；University of Leicester（莱斯特大学）；Central South University of Forestry and Technology（中南林业科技大学）；Massey University（梅西大学）
Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Ruili Wang, Jie Yang
Road extraction in remote sensing images is of great importance for a wide range of applications. Because of the complex background, and high density, most of the existing methods fail to accurately extract a road network that appears correct and complete. Moreover, they suffer from either insufficient training data or high costs of manual annotation. To address these problems, we introduce a new model to apply structured domain adaption for synthetic image generation and road segmentation. We incorporate a feature pyramid network into generative adversarial networks to minimize the difference between the source and target domains. A generator is learned to produce quality synthetic images, and the discriminator attempts to distinguish them. We also propose a feature pyramid network that improves the performance of the proposed model by extracting effective features from all the layers of the network for describing different scales objects. Indeed, a novel scale-wise architecture is introduced to learn from the multi-level feature maps and improve the semantics of the features. For optimization, the model is trained by a joint reconstruction loss function, which minimizes the difference between the fake images and the real ones. A wide range of experiments on three datasets prove the superior performance of the proposed approach in terms of accuracy and efficiency. In particular, our model achieves state-of-the-art 78.86 IOU on the Massachusetts dataset with 14.89M parameters and 86.78B FLOPs, with 4x fewer FLOPs but higher accuracy (+3.47% IOU) than the top performer among state-of-the-art approaches used in the evaluation.
Image super-resolution (SR) methods can generate remote sensing images with high spatial resolution without increasing the cost, thereby providing a feasible way to acquire high-resolution remote sensing images, which are difficult to obtain due to the high cost of acquisition equipment and complex weather. Clearly, image super-resolution is a severe ill-posed problem. Fortunately, with the development of deep learning, the powerful fitting ability of deep neural networks has solved this problem to some extent. In this paper, we propose a network based on the generative adversarial network (GA…
Nanjing University of Information Science and Technology（南京信息工程大学）；Beijing Electro-Mechanical Engineering Institute（北京机电工程研究所）；University of Extremadura（埃斯特雷马杜拉大学）
Convolutional neural networks (CNNs) have attracted increasing attention in the remote sensing community. Most CNNs only take the last fully-connected layers as features for the classification of remotely sensed images, discarding the other convolutional layer features which may also be helpful for classification purposes. In this paper, we propose a new adaptive deep pyramid matching (ADPM) model that takes advantage of the features from all of the convolutional layers for remote sensing image classification. To this end, the optimal fusing weights for different convolutional layers are learn…
Institute of Mountain Hazards and Environment, Chinese Academy of Sciences（中国科学院成都山地灾害与环境研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Bell Labs（贝尔实验室）；CAS Center for Excellence in Tibetan Plateau Earth Sciences, Chinese Academy of Sciences（中国科学院青藏高原地球科学卓越创新中心）；School of Economics and Management, Southwest Jiao Tong University（西南交通大学经济管理学院）
We consider the problem of segmentation and classification of high-resolution and hyperspectral remote sensing images. Unlike conventional natural (RGB) images, the inherent large scale and complex structures of remote sensing images pose major challenges such as spatial object distribution diversity and spectral information extraction when existing models are directly applied for image classification. In this study, we develop an attention-based pyramid network for segmentation and classification of remote sensing datasets. Attention mechanisms are used to develop the following modules: i) a …
TUBITAK Space Technologies Research Institute（土耳其科学技术研究理事会空间技术研究所）
Cloud detection from remotely observed data is a critical pre-processing step for various remote sensing applications. In particular, this problem becomes even harder for RGB color images, since there is no distinct spectral pattern for clouds, which is directly separable from the Earth surface. In this paper, we adapt a deep pyramid network (DPN) to tackle this problem. For this purpose, the network is enhanced with a pre-trained parameter model at the encoder layer. Moreover, the method is able to obtain accurate pixel-level segmentation and classification results from a set of noisy labeled…
Object counting, which aims to count the accurate number of object instances in images, has been attracting more and more attention. However, challenges such as large scale variation, complex background interference, and non-uniform density distribution greatly limit the counting accuracy, particularly striking in remote sensing imagery. To mitigate the above issues, this paper proposes a novel framework for dense object counting in remote sensing images, which incorporates a pyramidal scale module (PSM) and a global context module (GCM), dubbed PSGCNet, where PSM is used to adaptively capture…
Institute of Electronics, Chinese Academy of Sciences（中国科学院电子学研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Ship detection has been playing a significant role in the field of remote sensing for a long time but it is still full of challenges. The main limitations of traditional ship detection methods usually lie in the complexity of application scenarios, the difficulty of intensive object detection and the redundancy of detection region. In order to solve such problems above, we propose a framework called Rotation Dense Feature Pyramid Networks (R-DFPN) which can effectively detect ship in different scenes including ocean and port. Specifically, we put forward the Dense Feature Pyramid Network (DFPN…
Shanghai Jiao Tong University（上海交通大学）；GIPSA-lab, Université Grenoble Alpes, CNRS, Grenoble INP（格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔国立理工学院联合GIPSA实验室）；University of Iceland（冰岛大学）；University of Leicester（莱斯特大学）
Detection of objects is extremely important in various aerial vision-based applications. Over the last few years, the methods based on convolution neural networks have made substantial progress. However, because of the large variety of object scales, densities, and arbitrary orientations, the current detectors struggle with the extraction of semantically strong features for small-scale objects by a predefined convolution kernel. To address this problem, we propose the rotation equivariant feature image pyramid network (REFIPN), an image pyramid network based on rotation equivariance convolutio…
Shandong University（山东大学）；South China Normal University（华南师范大学）；China Telecom Corporation Limited（中国电信股份有限公司）；Nanjing University of Science and Technology（南京理工大学）；Tencent AI Lab（腾讯AI实验室）
City University of Hong Kong（香港城市大学）；Beijing Jiaotong University（北京交通大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）；Tianjin University（天津大学）
Arising from the various object types and scales, diverse imaging orientations, and cluttered backgrounds in optical remote sensing image (RSI), it is difficult to directly extend the success of salient object detection for nature scene image to the optical RSI. In this paper, we propose an end-to-end deep network called LV-Net based on the shape of network architecture, which detects salient objects from optical RSIs in a purely data-driven fashion. The proposed LV-Net consists of two key modules, i.e., a two-stream pyramid module (L-shaped module) and an encoder-decoder module with nested co…
Shanghai Jiao Tong University（上海交通大学）；École de technologie supérieure, Université du Québec（魁北克大学高等技术学院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）
Small-object detection is a challenging problem. In the last few years, the convolution neural networks methods have been achieved considerable progress. However, the current detectors struggle with effective features extraction for small-scale objects. To address this challenge, we propose image pyramid single-shot detector (IPSSD). In IPSSD, single-shot detector is adopted combined with an image pyramid network to extract semantically strong features for generating candidate regions. The proposed network can enhance the small-scale features from a feature pyramid network. We evaluated the pe…
Road surface extraction from remote sensing images using deep learning methods has achieved good performance, while most of the existing methods are based on fully supervised learning, which requires a large amount of training data with laborious per-pixel annotation. In this paper, we propose a scribble-based weakly supervised road surface extraction method named ScRoadExtractor, which learns from easily accessible scribbles such as centerlines instead of densely annotated road surface ground-truths. To propagate semantic information from sparse scribbles to unlabeled pixels, we introduce a r…
Object detection is a challenging task in remote sensing because objects only occupy a few pixels in the images, and the models are required to simultaneously learn object locations and detection. Even though the established approaches well perform for the objects of regular sizes, they achieve weak performance when analyzing small ones or getting stuck in the local minima (e.g. false object parts). Two possible issues stand in their way. First, the existing methods struggle to perform stably on the detection of small objects because of the complicated background. Second, most of the standard …
Wuhan University（武汉大学）；Lancaster University（兰卡斯特大学）；UK Centre for Ecology and Hydrology（英国生态与水文中心）
Semantic segmentation using fine-resolution remotely sensed images plays a critical role in many practical applications, such as urban planning, environmental protection, natural and anthropogenic landscape monitoring, etc. However, the automation of semantic segmentation, i.e., automatic categorization/labeling and segmentation is still a challenging task, particularly for fine-resolution images with huge spatial and spectral complexity. Addressing such a problem represents an exciting research field, which paves the way for scene-level landscape pattern analysis and decision making. In this …
Temple University（天普大学）；University of Pittsburgh（匹兹堡大学）；Toyota Research Institute（丰田研究院）；TuSimple（图森未来）
Pavement crack detection is a critical task for insuring road safety. Manual crack detection is extremely time-consuming. Therefore, an automatic road crack detection method is required to boost this progress. However, it remains a challenging task due to the intensity inhomogeneity of cracks and complexity of the background, e.g., the low contrast with surrounding pavements and possible shadows with similar intensity. Inspired by recent advances of deep learning in computer vision, we propose a novel network architecture, named Feature Pyramid and Hierarchical Boosting Network (FPHBN), for pa…
Centre for Research-based Innovation Visual Intelligence（可视化研究型创新中心）
Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based o…
Autonomous vehicles commonly rely on highly detailed birds-eye-view maps of their environment, which capture both static elements of the scene such as road layout as well as dynamic elements such as other cars and pedestrians. Generating these map representations on the fly is a complex multi-stage process which incorporates many important vision-based elements, including ground plane estimation, road segmentation and 3D object detection. In this work we present a simple, unified approach for estimating maps directly from monocular images using a single end-to-end deep learning architecture. F…
GLD-Road:A global-local decoding road network extraction model for remote sensing images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空信息研究所）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；Heilongjiang Geographic Information Engineering Institute（黑龙江地理信息工程研究所）
We propose an automated method to estimate a road segment's free-flow speed from overhead imagery and road metadata. The free-flow speed of a road segment is the average observed vehicle speed in ideal conditions, without congestion or adverse weather. Standard practice for estimating free-flow speeds depends on several road attributes, including grade, curve, and width of the right of way. Unfortunately, many of these fine-grained labels are not always readily available and are costly to manually annotate. To compensate, our model uses a small, easy to obtain subset of road features along wit…
DB SwinT: A Dual-Branch Swin Transformer Network for Road Extraction in Optical Remote Sensing Imagery
School of Information Science and Engineering（信息科学与工程学院）；Chongqing Jiaotong University（重庆交通大学）；College of Information Engineering（信息工程学院）；Inner Mongolia University of Technology（内蒙古科技大学）；Inner Mongolia Key Laboratory of Radar Technology and Application（内蒙古雷达技术与应用重点实验室）
A Large-Scale Remote Sensing Dataset and VLM-based Algorithm for Fine-Grained Road Hierarchy Classification
School of Geospatial Engineering and Science（地理工程与科学学院）；Laboratory of Intelligent Collaborative Computing（智能协同计算实验室）；United Nations University Institute in Macau（联合国澳门研究所）；Moganshan Geospatial Information Laboratory（莫干山地理信息实验室）
The binary segmentation of roads in very high resolution (VHR) remote sensing images (RSIs) has always been a challenging task due to factors such as occlusions (caused by shadows, trees, buildings, etc.) and the intra-class variances of road surfaces. The wide use of convolutional neural networks (CNNs) has greatly improved the segmentation accuracy and made the task end-to-end trainable. However, there are still margins to improve in terms of the completeness and connectivity of the results. In this paper, we consider the specific context of road extraction and present a direction-aware resi…
Wuhan University（武汉大学）；University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
Road segmentation from remote sensing images is a challenging task with wide ranges of application potentials. Deep neural networks have advanced this field by leveraging the power of large-scale labeled data, which, however, are extremely expensive and time-consuming to acquire. One solution is to use cheap available data to train a model and deploy it to directly process the data from a specific application domain. Nevertheless, the well-known domain shift (DS) issue prevents the trained model from generalizing well on the target domain. In this paper, we propose a novel stagewise domain ada…
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Institute of Technology（北京理工大学）；National and Kapodistrian University of Athens（雅典国立卡波季斯特里安大学）；China Aero Geophysical Survey and Remote Sensing Center for Natural Resources（中国自然资源航空物探遥感中心）；La Trobe University（乐卓博大学）
Extracting roads from high-resolution remote sensing images (HRSIs) is vital in a wide variety of applications, such as autonomous driving, path planning, and road navigation. Due to the long and thin shape as well as the shades induced by vegetation and buildings, small-sized roads are more difficult to discern. In order to improve the reliability and accuracy of small-sized road extraction when roads of multiple sizes coexist in an HRSI, an enhanced deep neural network model termed Dual-Decoder-U-Net (DDU-Net) is proposed in this paper. Motivated by the U-Net model, a small decoder is added …
LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Images
School of Medical Instruments, Shanghai University of Medicine and Health Sciences（上海健康医学院医疗器械学院）；School of Health Sciences and Engineering, University of Shanghai for Science and Technology（上海理工大学健康科学与工程学院）；Shanghai Children’s Medical Center, School of Medicine, Shanghai Jiao Tong University（上海交通大学医学院附属上海儿童医学中心）；School of Communication & Electronic Engineering, East China Normal University（华东师范大学通信与电子工程学院）；Institute of Seismology, China Earthquake Administration（中国地震局地震研究所）
In this work, we present a novel approach to learning an encoding of visual features into graph neural networks with the application on road network data. We propose an architecture that combines state-of-the-art vision backbone networks with graph neural networks. More specifically, we perform a road type classification task on an Open Street Map road network through encoding of satellite imagery using various ResNet architectures. Our architecture further enables fine-tuning and a transfer-learning approach is evaluated by pretraining on the NWPU-RESISC45 image classification dataset for rem…
Northeastern University（东北大学）；Shengjing Hospital, China Medical University（中国医科大学附属盛京医院）
In this paper, we propose a dual aggregation network to adaptively aggregate different information in infant brain MRI segmentation. More precisely, we added two modules based on 3D-UNet to better model information at different levels and locations. The dilated convolution pyramid downsampling module is mainly to solve the problem of loss of spatial information on the downsampling process, and it can effectively save details while reducing the resolution. The self-attention module can integrate the remote dependence on the feature maps in two dimensions of spatial and channel, effectively impr…
City University of Hong Kong（香港城市大学）；Beijing Jiaotong University（北京交通大学）
Despite the remarkable advances in visual saliency analysis for natural scene images (NSIs), salient object detection (SOD) for optical remote sensing images (RSIs) still remains an open and challenging problem. In this paper, we propose an end-to-end Dense Attention Fluid Network (DAFNet) for SOD in optical RSIs. A Global Context-aware Attention (GCA) module is proposed to adaptively capture long-range semantic context relationships, and is further embedded in a Dense Attention Fluid (DAF) structure that enables shallow attention cues flow into deep layers to guide the generation of high-leve…
Building extraction from aerial images has several applications in problems such as urban planning, change detection, and disaster management. With the increasing availability of data, Convolutional Neural Networks (CNNs) for semantic segmentation of remote sensing imagery has improved significantly in recent years. However, convolutions operate in local neighborhoods and fail to capture non-local features that are essential in semantic understanding of aerial images. In this work, we propose to improve building segmentation of different sizes by capturing long-range dependencies using context…
Recently, change detection (CD) of remote sensing images have achieved great progress with the advances of deep learning. However, current methods generally deliver incomplete CD regions and irregular CD boundaries due to the limited representation ability of the extracted visual features. To relieve these issues, in this work we propose a novel learning framework named Fully Transformer Network (FTN) for remote sensing image CD, which improves the feature extraction from a global view and combines multi-level visual features in a pyramid manner. More specifically, the proposed framework first…
Semantic Segmentation (SS) is a task to assign semantic label to each pixel of the images, which is of immense significance for autonomous vehicles, robotics and assisted navigation of vulnerable road users. It is obvious that in different application scenarios, different objects possess hierarchical importance and safety-relevance, but conventional loss functions like cross entropy have not taken the different levels of importance of diverse traffic elements into consideration. To address this dilemma, we leverage and re-design an importance-aware loss function, throwing insightful hints on h…
Most state-of-the-art approaches to road extraction from aerial images rely on a CNN trained to label road pixels as foreground and remainder of the image as background. The CNN is usually trained by minimizing pixel-wise losses, which is less than ideal to produce binary masks that preserve the road network's global connectivity. To address this issue, we introduce an Adversarial Learning (AL) strategy tailored for our purposes. A naive one would treat the segmentation network as a generator and would feed its output along with ground-truth segmentations to a discriminator. It would then trai…
Road extraction is an essential step in building autonomous navigation systems. Detecting road segments is challenging as they are of varying widths, bifurcated throughout the image, and are often occluded by terrain, cloud, or other weather conditions. Using just convolution neural networks (ConvNets) for this problem is not effective as it is inefficient at capturing distant dependencies between road segments in the image which is essential to extract road connectivity. To this end, we propose a Spatial and Interaction Space Graph Reasoning (SPIN) module which when plugged into a ConvNet per…
Road networks are the core infrastructure for connected and autonomous vehicles, but creating meaningful representations for machine learning applications is a challenging task. In this work, we propose to integrate remote sensing vision data into road network data for improved embeddings with graph neural networks. We present a segmentation of road edges based on spatio-temporal road and traffic characteristics, which allows to enrich the attribute set of road networks with visual features of satellite imagery and digital surface models. We show that both, the segmentation and the integration…
Remote sensor image object detection is an important technology for Earth observation, and is used in various tasks such as forest fire monitoring and ocean monitoring. Image object detection technology, despite the significant developments, is struggling to handle remote sensor images and small-scale objects, due to the limited pixels of small objects. Numerous existing studies have demonstrated that an effective way to promote small object detection is to introduce the spatial context. Meanwhile, recent researches for image classification have shown that spectral convolution operations can p…
Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images
The Hong Kong Polytechnic University（香港理工大学）；Wuhan University（武汉大学）；Changjiang Spatial Information Technology Engineering Co., Ltd.（长江空间信息技术工程有限公司）；Peng Cheng Laboratory（鹏城实验室）
Building and Road Recognition in Dense Urban Informal Settlements: A Dataset and Benchmark
Guangdong Provincial Project（广东省项目）；Guangzhou-HKUST(GZ) Joint Funding Program（广州-香港科技大学联合基金计划）；AI Research and Learning Base of Urban Culture Project（城市文化AI研究与学习基地项目）
Beihang University（北京航空航天大学）；University of Central Florida（中佛罗里达大学）；Lancaster University（兰卡斯特大学）；Huawei Technologies Co. Ltd.（华为技术有限公司）
Compression artifacts reduction (CAR) is a challenging problem in the field of remote sensing. Most recent deep learning based methods have demonstrated superior performance over the previous hand-crafted methods. In this paper, we propose an end-to-end one-two-one (OTO) network, to combine different deep models, i.e., summation and difference models, to solve the CAR problem. Particularly, the difference model motivated by the Laplacian pyramid is designed to obtain the high frequency information, while the summation model aggregates the low frequency information. We provide an in-depth inves…
Beihang University（北京航空航天大学）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
Estimating accurate number of interested objects from a given image is a challenging yet important task. Significant efforts have been made to address this problem and achieve great progress, yet counting number of ground objects from remote sensing images is barely studied. In this paper, we are interested in counting dense objects from remote sensing images. Compared with object counting in natural scene, this task is challenging in following factors: large scale variation, complex cluttered background and orientation arbitrariness. More importantly, the scarcity of data severely limits the …
Road extraction from aerial images has been a hot research topic in the field of remote sensing image analysis. In this letter, a semantic segmentation neural network which combines the strengths of residual learning and U-Net is proposed for road area extraction. The network is built with residual units and has similar architecture to that of U-Net. The benefits of this model is two-fold: first, residual units ease training of deep networks. Second, the rich skip connections within the network could facilitate information propagation, allowing us to design networks with fewer parameters howev…
In this paper we address three different aspects of semantic segmentation from remote sensor data using deep neural networks. Firstly, we focus on the semantic segmentation of buildings from remote sensor data and propose ICT-Net. The proposed network has been tested on the INRIA and AIRS benchmark datasets and is shown to outperform all other state of the art by more than 1.5% and 1.8% on the Jaccard index, respectively. Secondly, as the building classification is typically the first step of the reconstruction process, we investigate the relationship of the classification accuracy to the reco…
