Image translation with convolutional autoencoders has recently been used as an approach to multimodal change detection in bitemporal satellite images. A main challenge is the alignment of the code spaces by reducing the contribution of change pixels to the learning of the translation function. Many existing approaches train the networks by exploiting supervised information of the change areas, which, however, is not always available. We propose to extract relational pixel information captured by domain-specific affinity matrices at the input and use this to enforce alignment of the code spaces and reduce the impact of change pixels on the learning objective. A change prior is derived in an unsupervised fashion from pixel pair affinities that are comparable across domains. To achieve code space alignment we enforce that pixel with similar affinity relations in the input domains should be correlated also in code space. We demonstrate the utility of this procedure in combination with cycle consistency. The proposed approach are compared with state-of-the-art deep learning algorithms. Experiments conducted on four real datasets show the effectiveness of our methodology.
A$^{2}$-MAE: A spatial-temporal-spectral unified remote sensing pre-training method based on anchor-aware masked autoencoder
National Supercomputing Center in Shenzhen（深圳国家超级计算中心）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Sun Yat-Sen University（中山大学）；SenseTime Group（商汤集团）
Benefiting from the great success of deep learning in computer vision, CNN-based object detection methods have drawn significant attentions. Various frameworks have been proposed which show awesome and robust performance for a large range of datasets. However, for building detection in remote sensing images, buildings always pose a diversity of orientations which makes it a challenge for the application of off-the-shelf methods to building detection. In this work, we aim to integrate orientation regression into the popular axis-aligned bounding-box detection method to tackle this problem. To a…
Freq-RemoteVAR: Next-Frequency Autoregressive Modeling for Remote Sensing Change Detection
Zhejiang Lab（之江实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Changsha University of Science and Technology（长沙理工大学）；The Education University of Hong Kong（香港教育大学）；South China Agricultural University（华南农业大学）
Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学学院，柏林技术大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
In the U.S., corn is the most produced crop and has been an essential part of the American diet. To meet the demand for supply chain management and regional food security, accurate and timely large-scale corn yield prediction is attracting more attention in precision agriculture. Recently, remote sensing technology and machine learning methods have been widely explored for crop yield prediction. Currently, most county-level yield prediction models use county-level mean variables for prediction, ignoring much detailed information. Moreover, inconsistent spatial resolution between crop area and …
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Ulm University（乌尔姆大学）
For many countries like Russia, Canada, or the USA, a robust and detailed tree species inventory is essential to manage their forests sustainably. Since one can not apply unmanned aerial vehicle (UAV) imagery-based approaches to large-scale forest inventory applications, the utilization of machine learning algorithms on satellite imagery is a rising topic of research. Although satellite imagery quality is relatively low, additional spectral channels provide a sufficient amount of information for tree crown classification tasks. Assuming that tree crowns are detected already, we use embeddings …
Self-supervised learning aims to learn image feature representations without the usage of manually annotated labels. It is often used as a precursor step to obtain useful initial network weights which contribute to faster convergence and superior performance of downstream tasks. While self-supervision allows one to reduce the domain gap between supervised and unsupervised learning without the usage of labels, the self-supervised objective still requires a strong inductive bias to downstream tasks for effective transfer learning. In this work, we present our material and texture based self-supe…
rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement
Tianjin University（天津大学）；Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂工业大学（LUT大学））；Great Bay University（大湾区大学）；the Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
本文提出 rPPG-MAE 自监督预训练框架，通过线性重建任务和噪声不敏感策略挖掘生理信号的自相似先验，在多个 rPPG 数据集上超过现有自监督及 SOTA 监督方法。
Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电机工程与计算机科学学院）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Environmental Computational Science and Earth Observation Laboratory (ECEO)（环境计算科学与地球观测实验室）
Mixup is a powerful data augmentation method that interpolates between two or more examples in the input or feature space and between the corresponding target labels. Many recent mixup methods focus on cutting and pasting two or more objects into one image, which is more about efficient processing than interpolation. However, how to best interpolate images is not well defined. In this sense, mixup has been connected to autoencoders, because often autoencoders "interpolate well", for instance generating an image that continuously deforms into another. In this work, we revisit mixup from the int…
What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion
Shanghai Jiao Tong University（上海交通大学）；Alibaba Group（阿里巴巴集团）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Beihang University（北航）；Sun Yat-sen University（中山大学）；Nankai University（南开大学）；Shanghai AI Laboratory（上海人工智能实验室）
School of Computer & Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）
Learning a common latent embedding by aligning the latent spaces of cross-modal autoencoders is an effective strategy for Generalized Zero-Shot Classification (GZSC). However, due to the lack of fine-grained instance-wise annotations, it still easily suffer from the domain shift problem for the discrepancy between the visual representation of diversified images and the semantic representation of fixed attributes. In this paper, we propose an innovative autoencoder network by learning Aligned Cross-Modal Representations (dubbed ACMR) for GZSC. Specifically, we propose a novel Vision-Semantic Al…
SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretability
Department of Computer Science and Software Engineering (CSSE) Concordia University, Canada（计算机科学与软件工程系（CSSE）康科迪亚大学，加拿大）；Department of Electrical and Computer Engineering (ECE) Concordia University, Canada（电气与计算机工程系（ECE）康科迪亚大学，加拿大）
SatMamba: Development of Foundation Models for Remote Sensing Imagery Using State Space Models
University of Engineering and Technology, Vietnam National University（越南国家大学工程技术大学）；International Digital Earth Applied Science Research Center, Chubu University（中部大学国际数字地球应用科学研究中心）
Bosch Center for AI（博世人工智能中心）；UC Berkeley（加州大学伯克利分校）；University of Toronto（多伦多大学）；University of Amsterdam（阿姆斯特丹大学）
Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space. As labeled images are expensive, one direction is to augment the dataset by generating either images or image features. However, the former misses fine-grained details and the latter requires learning a mapping associated with class embeddings. In this work, we take feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by modality-specific aligned variational autoencoders. Thi…
Hyperspectral image analysis has become an important topic widely researched by the remote sensing community. Classification and segmentation of such imagery help understand the underlying materials within a scanned scene, since hyperspectral images convey a detailed information captured in a number of spectral bands. Although deep learning has established the state of the art in the field, it still remains challenging to train well-generalizing models due to the lack of ground-truth data. In this letter, we tackle this problem and propose an end-to-end approach to segment hyperspectral images…
CLIMB: Controllable Longitudinal Brain Image Generation using Mamba-based Latent Diffusion Model and Gaussian-aligned Autoencoder
Department of Artificial Intelligence Convergence, Chonnam National University（全南国立大学人工智能融合系）；Department of Nuclear Medicine, Chonnam National University（全南国立大学核医学系）；University of South Australia（南澳大利亚大学）；AdelaideMRI（阿德莱德MRI）；Electronics and Telecommunications Research Institute（电子电信研究所）
Research & Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Remote sensing image scene classification, which aims at labeling remote sensing images with a set of semantic categories based on their contents, has broad applications in a range of fields. Propelled by the powerful feature learning capabilities of deep neural networks, remote sensing image scene classification driven by deep learning has drawn remarkable attention and achieved significant breakthroughs. However, to the best of our knowledge, a comprehensive review of recent achievements regarding deep learning for scene classification of remote sensing images is still lacking. Considering t…
GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Space Applications Centre, ISRO（印度空间研究组织空间应用中心）；University of Trento（特伦托大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
In the remote sensing context spectral unmixing is a technique to decompose a mixed pixel into two fundamental representatives: endmembers and abundances. In this paper, a novel architecture is proposed to perform blind unmixing on hyperspectral images. The proposed architecture consists of convolutional layers followed by an autoencoder. The encoder transforms the feature space produced through convolutional layers to a latent space representation. Then, from these latent characteristics the decoder reconstructs the roll-out image of the monochrome image which is at the input of the architect…
Harbin Institute of Technology（哈尔滨工业大学）；Nanyang Technological University（南洋理工大学）
Hyperspectral image (HSI) classification is a hot topic in the remote sensing community. This paper proposes a new framework of spectral-spatial feature extraction for HSI classification, in which for the first time the concept of deep learning is introduced. Specifically, the model of autoencoder is exploited in our framework to extract various kinds of features. First we verify the eligibility of autoencoder by following classical spectral information based classification and use autoencoders with different depth to classify hyperspectral image. Further in the proposed framework, we combine …
Predicting wildfire spread is critical for land management and disaster preparedness. To this end, we present `Next Day Wildfire Spread,' a curated, large-scale, multivariate data set of historical wildfires aggregating nearly a decade of remote-sensing data across the United States. In contrast to existing fire data sets based on Earth observation satellites, our data set combines 2D fire data with multiple explanatory variables (e.g., topography, vegetation, weather, drought index, population density) aligned over 2D regions, providing a feature-rich data set for machine learning. To demonst…
RS3DBench: A Comprehensive Benchmark for 3D Spatial Perception in Remote Sensing
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Software College of Zhejiang University（浙江大学软件学院）；Hangzhou City University（杭州市大学）
Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）
Detecting aligned 3D keypoints is essential under many scenarios such as object tracking, shape retrieval and robotics. However, it is generally hard to prepare a high-quality dataset for all types of objects due to the ambiguity of keypoint itself. Meanwhile, current unsupervised detectors are unable to generate aligned keypoints with good coverage. In this paper, we propose an unsupervised aligned keypoint detector, Skeleton Merger, which utilizes skeletons to reconstruct objects. It is based on an Autoencoder architecture. The encoder proposes keypoints and predicts activation strengths of …
AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models
Space Applications Centre, ISRO（印度空间研究组织空间应用中心）；Indian Institute of Science Education and Research (IISER) Bhopal（印度科学教育与研究学院（博帕尔））；Indian Institute of Technology Bombay（印度理工学院孟买分校）
