Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.
Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging
Junia（朱尼亚学院）；Centrale Lille（里尔中央理工学院）；University of the Basque Country UPV/EHU（巴斯克大学 UPV/EHU）；Université Polytechnique Hauts-de-France（上法兰西理工大学）；Université de Lille（里尔大学）；Ho Chi Minh City Open University（胡志明市开放大学）
Veeva Systems（Veeva系统公司）；Lyft Inc（来福车公司）；Massachusetts Institute of Technology（麻省理工学院）
Semantic segmentation is in-demand in satellite imagery processing. Because of the complex environment, automatic categorization and segmentation of land cover is a challenging problem. Solving it can help to overcome many obstacles in urban planning, environmental engineering or natural landscape monitoring. In this paper, we propose an approach for automatic multi-class land segmentation based on a fully convolutional neural network of feature pyramid network (FPN) family. This network is consisted of pre-trained on ImageNet Resnet50 encoder and neatly developed decoder. Based on validation …
Department of Computing Science, University of Alberta（阿尔伯塔大学计算科学系）
CNNs, initially inspired by human vision, differ in a key way: they sample uniformly, rather than with highest density in a focal point. For very large images, this makes training untenable, as the memory and computation required for activation maps scales quadratically with the side length of an image. We propose an image pyramid based approach that extracts narrow glimpses of the of the input image and iteratively refines them to accomplish regression tasks. To assist with high-accuracy regression, we introduce a novel intermediate representation we call 'spatialized features'. Our approach …
Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）
Most salient object detection approaches use U-Net or feature pyramid networks (FPN) as their basic structures. These methods ignore two key problems when the encoder exchanges information with the decoder: one is the lack of interference control between them, the other is without considering the disparity of the contributions of different encoder blocks. In this work, we propose a simple gated network (GateNet) to solve both issues at once. With the help of multilevel gate units, the valuable context information from the encoder can be optimally transmitted to the decoder. We design a novel g…
The University of Hong Kong（香港大学）；Modontics (Hong Kong) Limited（Modontics（香港）有限公司）
Marking anatomical landmarks in cephalometric radiography is a critical operation in cephalometric analysis. Automatically and accurately locating these landmarks is a challenging issue because different landmarks require different levels of resolution and semantics. Based on this observation, we propose a novel attentive feature pyramid fusion module (AFPF) to explicitly shape high-resolution and semantically enhanced fusion features to achieve significantly higher accuracy than existing deep learning-based methods. We also combine heat maps and offset maps to perform pixel-wise regression-vo…
The production of thematic maps depicting land cover is one of the most common applications of remote sensing. To this end, several semantic segmentation approaches, based on deep learning, have been proposed in the literature, but land cover segmentation is still considered an open problem due to some specific problems related to remote sensing imaging. In this paper we propose a novel approach to deal with the problem of modelling multiscale contexts surrounding pixels of different land cover categories. The approach leverages the computation of a heteroscedastic measure of uncertainty when …
Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）
The next generation of Mars rotorcrafts requires on-board autonomous hazard avoidance landing. To this end, this work proposes a system that performs continuous multi-resolution height map reconstruction and safe landing spot detection. Structure-from-Motion measurements are aggregated in a pyramid structure using a novel Optimal Mixture of Gaussians formulation that provides a comprehensive uncertainty model. Our multiresolution pyramid is built more efficiently and accurately than past work by decoupling pyramid filling from the measurement updates of different resolutions. To detect the saf…
University of Basel（巴塞尔大学）；University Hospital Basel（巴塞尔大学医院）；MIAC AG
We present an automated method for localizing an anatomical landmark in three-dimensional medical images. The method combines two recurrent neural networks in a coarse-to-fine approach: The first network determines a candidate neighborhood by analyzing the complete given image volume. The second network localizes the actual landmark precisely and accurately in the candidate neighborhood. Both networks take advantage of multi-dimensional gated recurrent units in their main layers, which allow for high model complexity with a comparatively small set of parameters. We localize the medullopontine …
Pedestrian detection is an essential task in autonomous driving research. In addition to typical color images, thermal images benefit the detection in dark environments. Hence, it is worthwhile to explore an integrated approach to take advantage of both color and thermal images simultaneously. In this paper, we propose a novel approach to fuse color and thermal sensors using deep neural networks (DNN). Current state-of-the-art DNN object detectors vary from two-stage to one-stage mechanisms. Two-stage detectors, like Faster-RCNN, achieve higher accuracy, while one-stage detectors such as Singl…
Digital Elevation models (DEM) are images having terrain information embedded into them. Using cognitive mapping concepts for DEM registration, has evolved from this basic idea of using the mapping between the space to objects and defining their relationships to form the basic landmarks that need to be marked, stored and manipulated in and about the environment or other candidate environments, namely, in our case, the DEMs. The progressive two-level encapsulation of methods of geo-spatial cognition includes landmark knowledge and layout knowledge and can be useful for DEM registration. Space-b…
Fusing a sequence of perfectly aligned images captured at various exposures, has shown great potential to approach High Dynamic Range (HDR) imaging by sensors with limited dynamic range. However, in the presence of large motion of scene objects or the camera, mis-alignment is almost inevitable and leads to the notorious ``ghost'' artifacts. Besides, factors such as the noise in the dark region or color saturation in the over-bright region may also fail to fill local image details to the HDR image. This paper provides a novel multi-exposure fusion model based on Swin Transformer. Particularly, …
Versatile and Efficient Medical Image Super-Resolution Via Frequency-Gated Mamba
Faculty of Engineering and Information Technology, University of Technology Sydney, Australia（工程与信息技术学院，悉尼技术大学，澳大利亚）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所，中国科学院，中国）；College of Computer Science and Technology, Qingdao University, China（计算机科学与技术学院，青岛大学，中国）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology, China（计算机科学与控制工程学院，深圳先进技术大学，中国）
Carnegie Mellon University（卡内基梅隆大学）；IBM Research AI（IBM研究院人工智能）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Google AI（谷歌人工智能）；UMass Amherst（马萨诸塞大学阿默斯特分校）；Stevens Institute of Technology（史蒂文斯理工学院）
In this paper, we present a simple and parameter-efficient drop-in module for one-stage object detectors like SSD when learning from scratch (i.e., without pre-trained models). We call our module GFR (Gated Feature Reuse), which exhibits two main advantages. First, we introduce a novel gate-controlled prediction strategy enabled by Squeeze-and-Excitation to adaptively enhance or attenuate supervision at different scales based on the input object size. As a result, our model is more effective in detecting diverse sizes of objects. Second, we propose a feature-pyramids structure to squeeze rich …
Wuhan University（武汉大学）；Lancaster University（兰卡斯特大学）；UK Centre for Ecology and Hydrology（英国生态与水文中心）
Semantic segmentation using fine-resolution remotely sensed images plays a critical role in many practical applications, such as urban planning, environmental protection, natural and anthropogenic landscape monitoring, etc. However, the automation of semantic segmentation, i.e., automatic categorization/labeling and segmentation is still a challenging task, particularly for fine-resolution images with huge spatial and spectral complexity. Addressing such a problem represents an exciting research field, which paves the way for scene-level landscape pattern analysis and decision making. In this …
Spatial Gated Multi-Layer Perceptron for Land Use and Land Cover Mapping
Simon Fraser University（西蒙菲莎大学）；Alipurduar Government Engineering and Management College（阿里布尔杜阿尔政府工程与管理学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Lancaster University（兰卡斯特大学）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；Institute of Advanced Research in Artificial Intelligence (IARAI)（人工智能高级研究所）
Facial expression recognition (FER) plays an important role in a variety of real-world applications such as human-computer interaction. POSTER achieves the state-of-the-art (SOTA) performance in FER by effectively combining facial landmark and image features through two-stream pyramid cross-fusion design. However, the architecture of POSTER is undoubtedly complex. It causes expensive computational costs. In order to relieve the computational pressure of POSTER, in this paper, we propose POSTER++. It improves POSTER in three directions: cross-fusion, two-stream, and multi-scale feature extracti…
Facial landmark localization plays an important role in face recognition and analysis applications. In this paper, we give a brief introduction to a coarse-to-fine pipeline with neural networks and sequential regression. First, a global convolutional network is applied to the holistic facial image to give an initial landmark prediction. A pyramid of multi-scale local image patches is then cropped to feed to a new network for each landmark to refine the prediction. As the refinement network outputs a more accurate position estimation than the input, such procedure could be repeated several time…
Self-supervised prediction is a powerful mechanism to learn representations that capture the underlying structure of the data. Despite recent progress, the self-supervised video prediction task is still challenging. One of the critical factors that make the task hard is motion segmentation, which is segmenting individual objects and the background and estimating their motion separately. In video prediction, the shape, appearance, and transformation of each object should be understood only by predicting the next frame in pixel space. To address this task, we propose a novel end-to-end learnable…
Spatial-Frequency Gated Swin Transformer for Cross-Sensor Remote Sensing Super-Resolution
Space Applications Centre, ISRO, Ahmedabad, India（印度航天研究组织阿赫迈德亚布德研究中心）；Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, India（印度理工学院孟买资源工程研究学院）；New L J Institute of Engineering and Technology, Ahmedabad, India（阿赫迈德亚布德新LJ工程与技术学院）；Pandit Deendayal Energy University, Gandhinagar, India（潘迪特·德恩达尔能源大学）；GLS University, Ahmedabad, India（阿赫迈德亚布德GLS大学）
The University of Hong Kong（香港大学）；ShanghaiTech University（上海科技大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Detecting 3D landmarks on cone-beam computed tomography (CBCT) is crucial to assessing and quantifying the anatomical abnormalities in 3D cephalometric analysis. However, the current methods are time-consuming and suffer from large biases in landmark localization, leading to unreliable diagnosis results. In this work, we propose a novel Structure-Aware Long Short-Term Memory framework (SA-LSTM) for efficient and accurate 3D landmark detection. To reduce the computational burden, SA-LSTM is designed in two stages. It first locates the coarse landmarks via heatmap regression on a down-sampled CB…
DFPENet-geology: A Deep Learning Framework for High Precision Recognition and Segmentation of Co-seismic Landslides
Institute of Mountain Hazards and Environment, Chinese Academy of Sciences（中国科学院成都山地灾害与环境研究所）；CAS Center for Excellence in Tibetan Plateau Earth Sciences, Chinese Academy of Sciences（中国科学院青藏高原地球科学卓越创新中心）；University of Chinese Academy of Sciences（中国科学院大学）；Chengdu University of Technology（成都理工大学）；Sichuan Geomatics Center（四川省测绘地理信息中心）
Microwave Remote Sensing Lab Center of Studies in Resources Engineering（微波遥感实验室 资源工程研究中心）；Centre of Machine Intelligence and Data Science（智能与数据科学中心）；Department of Electrical Engineering（电气工程系）；School of Mathematics and Statistics（数学与统计学学院）；Center of Studies in Resources Engineering（资源工程研究中心）
LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition
School of Airspace and Engineering, Shandong University（山东大学航空航天与工程学院）；School of Computer Science and Technology, Shandong University of Finance and Economics（山东财经大学计算机科学与技术学院）；School of Psychology and Neuroscience, University of Glasgow（格拉斯哥大学心理学与神经科学学院）；Faculty of Information Science and Engineering, Ocean University of China（中国海洋大学信息科学与工程学院）
Visual place recognition is one of the essential and challenging problems in the fields of robotics. In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments. We achieve this by first designing a novel deep learning architecture to generate the static semantic segmentation and recover the static image directly from the corresponding dynamic image. We then innovatively leverage the spatial-pyramid-matching model to encode the static semantic segmentation into fea…
URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation
University of Technology Sydney（悉尼科技大学）；Nanjing University of Science and Technology（南京理工大学）；Honor Device Co., Ltd.（荣耀终端有限公司）；Shanghai Institute of Microsystem and Information Technology, CAS（中国科学院上海微系统与信息技术研究所）
Multi-Modal Semantic Segmentation of Electrolyzer Components for Sustainable Hydrogen Technologies: A Dual-Branch Deep Learning Approach
Applied Artificial Intelligence and INtelligent Systems (AAIINS) Laboratory（应用人工智能与智能系统（AAIINS）实验室）；University of Scholars（学者大学）；Murdoch University（莫道克大学）；Charles Darwin University（查尔斯达尔文大学）
