We propose in this article to build up a collaboration between a deep neural network and a human in the loop to swiftly obtain accurate segmentation maps of remote sensing images. In a nutshell, the agent iteratively interacts with the network to correct its initially flawed predictions. Concretely, these interactions are annotations representing the semantic labels. Our methodological contribution is twofold. First, we propose two interactive learning schemes to integrate user inputs into deep neural networks. The first one concatenates the annotations with the other network's inputs. The second one uses the annotations as a sparse ground-truth to retrain the network. Second, we propose an active learning strategy to guide the user towards the most relevant areas to annotate. To this purpose, we compare different state-of-the-art acquisition functions to evaluate the neural network uncertainty such as ConfidNet, entropy or ODIN. Through experiments on three remote sensing datasets, we show the effectiveness of the proposed methods. Notably, we show that active learning based on uncertainty estimation enables to quickly lead the user towards mistakes and that it is thus relevant to guide the user interventions.
VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Information Science and Technology, Beijing Forestry University（北京林业大学信息科学与技术学院）
Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis
Beihang University（北京航空航天大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Astronautics, Beihang University（北京航空航天大学宇航学院）；Shen Yuan Honors College of Beihang University（北京航空航天大学沈元荣誉学院）
IBM Research AI（IBM人工智能研究院）；Fusemachines Inc.（福赛机器公司）
Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement l…
Deep learning-based interactive segmentation in remote sensing
National Center for Ecological Analysis and Synthesis, University of California, Santa Barbara（加州大学圣塔芭芭拉分校国家生态分析与综合中心）；University of Idaho（爱达荷大学）；Fujian Agriculture and Forestry University（福建农林大学）；Bren School of Environmental Science and Management, University of California, Santa Barbara（加州大学圣塔芭芭拉分校布伦环境科学与管理学院）
Govt. Arts College, Melur（梅卢尔政府艺术学院）；Sri Meenakshi Govt. Arts College for Women, Madurai（马杜赖斯里米纳克希政府女子艺术学院）
Remote sensing image classification can be performed in many different ways to extract meaningful features. One common approach is to perform edge detection. A second approach is to try and detect whole shapes, given the fact that these shapes usually tend to have distinctive properties such as object foreground or background. To get optimal results, these two approaches can be combined. This paper adopts a combinatorial optimization method to adaptively select threshold based features to improve remote sensing image. Feature selection is an important combinatorial optimization problem in the …
Technical University of Munich（慕尼黑工业大学）；Madras Institute of Technology, Anna University（安娜大学马德拉斯理工学院）
A robot operating in unstructured environments must be able to discriminate between different grasping styles depending on the prospective manipulation task. Having a system that allows learning from remote non-expert demonstrations can very feasibly extend the cognitive skills of a robot for task-oriented grasping. We propose a novel two-step framework towards this aim. The first step involves grasp area estimation by segmentation. We receive grasp area demonstrations for a new task via interactive segmentation, and learn from these few demonstrations to estimate the required grasp area on an…
Tree-GPT: Modular Large Language Model Expert System for Forest Remote Sensing Image Understanding and Interactive Analysis
Shenzhen University（深圳大学）；Research Institute for Smart Cities（智慧城市研究院）；State Key Laboratory of Subtropical Building and Urban Science（亚热带建筑与城市科学国家重点实验室）；Guangdong–Hong Kong-Macau Joint Laboratory for Smart Cities（粤港澳智慧城市联合实验室）
Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VISITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) identify when to interact vs. navigate via imitation learning …
Towards Comprehensive Interactive Change Understanding in Remote Sensing: A Large-scale Dataset and Dual-granularity Enhanced VLM
Research Center for Space Computing System, Zhejiang Lab（浙江省实验室空间计算系统研究中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Cyber Science and Engineering, Zhengzhou University（郑州大学网络科学与工程学院）；Liaoning University of Technology（辽宁科技学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University（武汉测绘遥感与信息工程国家重点实验室（LIESMARS））；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
March in Chat: Interactive Prompting for Remote Embodied Referring Expression
Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
A Spatial-Spectral-Frequency Interactive Network for Multimodal Remote Sensing Classification
Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）；School of Information and Electronics, Beijing Institute of Technology（北京理工大学信息与电子学院）；Beijing Key Laboratory of Fractional Signals and Systems, Beijing Institute of Technology（北京理工大学分数域信号与系统北京市重点实验室）；School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Key Laboratory of Collaborative Intelligent Systems of Ministry of Education, Xidian University（西安电子科技大学教育部协同智能系统重点实验室）；Academy of Artificial Intelligence, Inner Mongolia Normal University（内蒙古师范大学人工智能研究院）
