Novel view synthesis of remote sensing scenes is of great significance for scene visualization, human-computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for remote sensing images due to its high complexity, view sparsity and limited view-perspective variations. In this paper, we propose a novel remote sensing view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of remote sensing images, we represent the 3D space by combining implicit multiplane images (MPI) representation and deep neural networks. The 3D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multi-view input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as Implicit Multiplane Images (ImMPI). To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of remote sensing 3D scenes and proposed a neural network based Prior extractor to accelerate the optimization process. In addition, we propose a new dataset for remote sensing novel view synthesis with multi-view real-world google earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design. Our dataset and code can be found at https://github.com/wyc-Chang/ImMPI
GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing
KTH Royal Institute of Technology（瑞典皇家理工学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Technical University of Munich（慕尼黑工业大学）
We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. Unlike traditional MPI that uses a set of simple RGB$α$ planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as …
Lightweight Multiplane Images Network for Real-Time Stereoscopic Conversion from Planar Video
Hefei University of Technology（合肥工业大学）；Peng Cheng National Laboratory（鹏城国家实验室）；Anhui University（安徽大学）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）
P2PFormer: A Primitive-to-polygon Method for Regular Building Contour Extraction from Remote Sensing Images
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；College of Oceanography and Space Informatics, China University of Petroleum (East China)（中国石油大学（华东）海洋与空间信息学院）；School of Surveying and Geoinformation Engineering, East China University of Technology（东华理工大学测绘与地理信息工程学院）
FreqPhys: Repurposing Implicit Physiological Frequency Prior for Robust Remote Photoplethysmography
Hefei University of Technology（合肥工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Science and Technology Beijing（北京科技大学）；Great Bay University（大湾区大学）
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
New York University（纽约大学）；NYU Abu Dhabi（纽约大学阿布扎比分校）
Recent studies have shown the benefits of using additional elevation data (e.g., DSM) for enhancing the performance of the semantic segmentation of aerial images. However, previous methods mostly adopt 3D elevation information as additional inputs. While in many real-world applications, one does not have the corresponding DSM information at hand and the spatial resolution of acquired DSM images usually do not match the aerial images. To alleviate this data constraint and also take advantage of 3D elevation information, in this paper, we introduce a geometry-aware segmentation model that achiev…
PhyDAE: Physics-Guided Degradation-Adaptive Experts for All-in-One Remote Sensing Image Restoration
School of Electronic and Information Engineering, Harbin Institute of Technology（电子与信息工程学院，哈尔滨工业大学）；Heilongjiang Province Key Laboratory of Space-Air-Ground Integrated Intelligent Remote Sensing（黑龙江省空间-空天地一体化智能遥感重点实验室）
Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion
School of Computer Technology and Application, Qinghai University（青海大学计算机技术与应用学院）；Intelligent Computing and Application Laboratory of Qinghai Province, Qinghai University（青海省智能计算与应用实验室，青海大学）；Key Lab of Big Data & Artificial Intelligence in Transportation (Ministry of Education), School of Computer Science & Technology, Beijing Jiaotong University（交通运输大数据与人工智能重点实验室（教育部），北京交通大学计算机科学与技术学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Key Laboratory of Pervasive Computing, Ministry of Education（教育部普适计算重点实验室）
University of Waterloo（滑铁卢大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Novel view synthesis is a long-standing problem that revolves around rendering frames of scenes from novel camera viewpoints. Volumetric approaches provide a solution for modeling occlusions through the explicit 3D representation of the camera frustum. Multi-plane Images (MPI) are volumetric methods that represent the scene using front-parallel planes at distinct depths but suffer from depth discretization leading to a 2.D scene representation. Another line of approach relies on implicit 3D scene representations. Neural Radiance Fields (NeRF) utilize neural networks for encapsulating the conti…
EcoVision Lab - Photogrammetry and Remote Sensing, ETH Zürich（苏黎世联邦理工学院生态视觉实验室 - 摄影测量与遥感）
We investigate active learning in the context of deep neural network models for change detection and map updating. Active learning is a natural choice for a number of remote sensing tasks, including the detection of local surface changes: changes are on the one hand rare and on the other hand their appearance is varied and diffuse, making it hard to collect a representative training set in advance. In the active learning setting, one starts from a minimal set of training examples and progressively chooses informative samples that are annotated by a user and added to the training set. Hence, a …
Contributions to Label-Efficient Learning in Computer Vision and Remote Sensing
Université Bretagne Sud（南布列塔尼大学）；UMR IRISA（IRISA联合研究单位）；CentraleSupélec（中央理工-高等电力学院）；INRIA（法国国家信息与自动化研究所）；Technische Universität Berlin（柏林工业大学）；Université Savoie Mont Blanc（萨瓦勃朗峰大学）；Université de Caen Normandie（卡昂诺曼底大学）
Dynamic Dictionary Learning for Remote Sensing Image Segmentation
School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；Department of Computer Technology and Applications, Qinghai University（青海大学计算机技术与应用系）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Deep learning-based algorithms have greatly improved the performance of remote sensing image (RSI) super-resolution (SR). However, increasing network depth and parameters cause a huge burden of computing and storage. Directly reducing the depth or width of existing models results in a large performance drop. We observe that the SR difficulty of different regions in an RSI varies greatly, and existing methods use the same deep network to process all regions in an image, resulting in a waste of computing resources. In addition, existing SR methods generally predefine integer scale factors and ca…
Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR)（德国航空航天中心遥感技术研究所）；Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM)（慕尼黑工业大学地球观测信号处理实验室）；Department of Geography, University of Zurich（苏黎世大学地理系）；Laboratory of GeoInformation Science and Remote Sensing, Wageningen University of Research（瓦赫宁根大学地理信息科学与遥感实验室）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Institute of Computer Graphics and Vision, TU Graz（格拉茨技术大学计算机图形与视觉研究所）
Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensi…
InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition
Aerospace Information Research Institute, Chinese Academy of Sciences（航天信息研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai Jiao Tong University（上海交通大学）；University of Wollongong（沃林戈大学）；SAIS, Shanghai Jiao Tong University（上海交通大学国际关系学院）
InterPruner: Interactive Structured Pruning via Taylor-Implicit Criterion and Language-Prior Modulator for Multimodal Object Detection
Beijing University of Technology（北京工业大学）；Southwest University（西南大学）；China University of Geosciences Wuhan（中国地质大学（武汉））；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Yunnan Normal University（云南师范大学）；Beijing Forestry University（北京林业大学）；Ghent University（根特大学）
Nowadays, there is an explosive growth of screen contents due to the wide application of screen sharing, remote cooperation, and online education. To match the limited terminal bandwidth, high-resolution (HR) screen contents may be downsampled and compressed. At the receiver side, the super-resolution (SR) of low-resolution (LR) screen content images (SCIs) is highly demanded by the HR display or by the users to zoom in for detail observation. However, image SR methods mostly designed for natural images do not generalize well for SCIs due to the very different image characteristics as well as …
S2C: Learning Noise-Resistant Differences for Unsupervised Change Detection in Multimodal Remote Sensing Images
Information Engineering University（信息工程大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）
Masks Can Talk: Extracting Structured Text Information from Single-Modal Images for Remote Sensing Change Detection
Zhejiang University（浙江大学）；Harbin Institute of Technology（哈尔滨工业大学）；Harbin Institute of Technology Suzhou Research Institute（哈尔滨工业大学苏州研究院）；North China University of Water Resources and Electric Power（华北水利水电大学）；University of Auckland（奥克兰大学）
IR275K: A Benchmark for Infrared Multi-Frame Super-Resolution Toward Efficient Remote Sensing
Hangzhou Institute for Advanced Study（杭州高等研究院）；Shanghai Institute of Technical Physics of the Chinese Academy of Sciences（中国科学院上海技术物理研究所）；University of Chinese Academy of Sciences（中国科学院大学）
GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation
School of Internet, Anhui University（安徽大学互联网学院）；National Engineering Research Center for Agro-Ecological Big Data Analysis & Application, Anhui University（安徽大学农业生态大数据分析与应用国家工程研究中心）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
School of Geodesy and Geomatics, Wuhan University（武汉大学测绘学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Recently, satellites with high temporal resolution have fostered wide attention in various practical applications. Due to limitations of bandwidth and hardware cost, however, the spatial resolution of such satellites is considerably low, largely limiting their potentials in scenarios that require spatially explicit information. To improve image resolution, numerous approaches based on training low-high resolution pairs have been proposed to address the super-resolution (SR) task. Despite their success, however, low/high spatial resolution pairs are usually difficult to obtain in satellites wit…
Nowadays, online screen sharing and remote cooperation are becoming ubiquitous. However, the screen content may be downsampled and compressed during transmission, while it may be displayed on large screens or the users would zoom in for detail observation at the receiver side. Therefore, developing a strong and effective screen content image (SCI) super-resolution (SR) method is demanded. We observe that the weight-sharing upsampler (such as deconvolution or pixel shuffle) could be harmful to sharp and thin edges in SCIs, and the fixed scale upsampler makes it inflexible to fit screens with va…
Continuous Representation Methods, Theories, and Applications: An Overview and Perspectives
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710000, China（西安交通大学数学与统计学院）；School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu 610000, China（电子科技大学数学科学学院）
Remote SAMsing: From Segment Anything to Segment Everything
Department of Electrical Engineering, University of Bras\' i lia, Bras\' i lia, Brazil；Department of Geography, University of Bras\' i lia, Bras\' i lia, Brazil
