通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2402.06475cs.CV
用于遥感图像描述与检索的大型语言模型
Large Language Models for Captioning and Retrieving Remote Sensing Images
INESC-ID, Instituto Superior Técnico, University of Lisbon（里斯本大学高等技术学院INESC-ID研究所）；Universidade NOVA de Lisboa（新里斯本大学）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
João Daniel Silva, João Magalhães, Devis Tuia, Bruno Martins
Image captioning and cross-modal retrieval are examples of tasks that involve the joint analysis of visual and linguistic information. In connection to remote sensing imagery, these tasks can help non-expert users in extracting relevant Earth observation information for a variety of applications. Still, despite some previous efforts, the development and application of vision and language models to the remote sensing domain have been hindered by the relatively small size of the available datasets and models used in previous studies. In this work, we propose RS-CapRet, a Vision and Language method for remote sensing tasks, in particular image captioning and text-image retrieval. We specifically propose to use a highly capable large decoder language model together with image encoders adapted to remote sensing imagery through contrastive language-image pre-training. To bridge together the image encoder and language decoder, we propose training simple linear layers with examples from combining different remote sensing image captioning datasets, keeping the other parameters frozen. RS-CapRet can then generate descriptions for remote sensing images and retrieve images from textual descriptions, achieving SOTA or competitive performance with existing methods. Qualitative results illustrate that RS-CapRet can effectively leverage the pre-trained large language model to describe remote sensing images, retrieve them based on different types of queries, and also show the ability to process interleaved sequences of images and text in a dialogue manner.
Stanford University（斯坦福大学）；Meta AI；University of Surrey（萨里大学）
Multimodal tasks in the fashion domain have significant potential for e-commerce, but involve challenging vision-and-language learning problems - e.g., retrieving a fashion item given a reference image plus text feedback from a user. Prior works on multimodal fashion tasks have either been limited by the data in individual benchmarks, or have leveraged generic vision-and-language pre-training but have not taken advantage of the characteristics of fashion data. Additionally, these works have mainly been restricted to multimodal understanding tasks. To address these gaps, we make two key contrib…
Northwestern Polytechnical University（西北工业大学）；Center for OPTical IMagery Analysis and Learning (OPTIMAL)（光学影像分析与学习中心（OPTIMAL））
The spatial attention is a straightforward approach to enhance the performance for remote sensing image captioning. However, conventional spatial attention approaches consider only the attention distribution on one fixed coarse grid, resulting in the semantics of tiny objects can be easily ignored or disturbed during the visual feature extraction. Worse still, the fixed semantic level of conventional spatial attention limits the image understanding in different levels and perspectives, which is critical for tackling the huge diversity in remote sensing images. To address these issues, we propo…
Change Captioning in Remote Sensing: Evolution to SAT-Cap -- A Single-Stage Transformer Approach
Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Technical University of Munich（慕尼黑工业大学）；Lancaster University（兰卡斯特大学）
Enhancing Perception of Key Changes in Remote Sensing Image Change Captioning
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University（武汉大学计算机学院国家多媒体软件工程技术研究中心）；Hubei Luojia Laboratory（湖北珞珈实验室）；School of Urban Design, Wuhan University（武汉大学城市设计学院）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）
SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning
Austrian Institute of Technology（奥地利技术研究院）；University of Technology of Troyes（图卢兹技术大学）；Ho Chi Minh University of Technology（胡志明技术大学）；Ton Duc Thang University（-ton Duc Thang大学）
A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning
School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））；Northwest Institute of Nuclear Technology（西北核技术研究所）
Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenization
Anhui Province Key Laboratory of Wetland Ecosystem Protection and Restoration, Anhui University（安徽大学安徽省湿地生态保护与修复重点实验室）；School of Resources and Environmental Engineering, Anhui University（安徽大学资源与环境工程学院）；Shanghai Ubiquitous Navigation Technology Co. Ltd（上海普适导航科技股份有限公司）；The 54th Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第五十四研究所）
Semantic-Spatial Feature Fusion with Dynamic Graph Refinement for Remote Sensing Image Captioning
School of Computer Science and Technology, Wuhan University of Science and Technology（武汉科技大学计算机科学与技术学院）；School of Information Science and Engineering, Wuhan University of Science and Technology（武汉科技大学信息科学与工程学院）
BTCChat: Advancing Remote Sensing Bi-temporal Change Captioning with Multimodal Large Language Model
State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Aerospace Information Research Institute（航天信息研究所）；Chinese Academy of Sciences（中国科学院）；School of Geographical Sciences（地理科学学院）；Hunan Normal University（湖南师范大学）
Changes to Captions: An Attentive Network for Remote Sensing Change Captioning
Institute of Advanced Research in Artificial Intelligence（人工智能高级研究院）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（亥姆霍兹德累斯顿罗森多夫研究中心）；Helmholtz Institute Freiberg for Resource Technology (HIF)（亥姆霍兹弗赖贝格资源技术研究所）
RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（人工智能安全国家重点实验室，计算技术研究所，中国科学院，北京，中国）；Great Bay University, Dongguan, China（东莞Great Bay大学，中国）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳学院，中国）；Dongguan Key Laboratory for Intelligence and Information Technology, Dongguan, China（东莞智能与信息科技重点实验室，中国）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Technical University of Munich（慕尼黑工业大学）
We deal with the problem of generating textual captions from optical remote sensing (RS) images using the notion of deep reinforcement learning. Due to the high inter-class similarity in reference sentences describing remote sensing data, jointly encoding the sentences and images encourages prediction of captions that are semantically more precise than the ground truth in many cases. To this end, we introduce an Actor Dual-Critic training strategy where a second critic model is deployed in the form of an encoder-decoder RNN to encode the latent information corresponding to the original and gen…
Remote sensing image retrieval (RSIR) is the process of ranking database images depending on the degree of similarity compared to the query image. As the complexity of RSIR increases due to the diversity in shooting range, angle, and location of remote sensors, there is an increasing demand for methods to address these issues and improve retrieval performance. In this work, we introduce a novel method for retrieving aerial images by merging group convolution with attention mechanism and metric learning, resulting in robustness to rotational variations. For refinement and emphasis on important …
Deep neural networks (DNNs) have been recently found popular for image captioning problems in remote sensing (RS). Existing DNN based approaches rely on the availability of a training set made up of a high number of RS images with their captions. However, captions of training images may contain redundant information (they can be repetitive or semantically similar to each other), resulting in information deficiency while learning a mapping from the image domain to the language domain. To overcome this limitation, in this paper, we present a novel Summarization Driven Remote Sensing Image Captio…
Good Representation, Better Explanation: Role of Convolutional Neural Networks in Transformer-Based Remote Sensing Image Captioning
Department of Computer Science and Engineering（计算机科学与工程系）；Indian Institute of Technology, Roorkee（印度理工学院拉胡尔分校）；Department of Mechanical Engineering（机械工程系）；Indian Institute of Technology, BHU（印度理工学院BHU分校）；Department of Information Technology（信息技术系）；National Institute of Technology, Srinagar（国家理工学院斯里 Nagar分校）
HiSem: Hierarchical Semantic Disentangling for Remote Sensing Image Change Captioning
College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）；Research Center for Spatiotemporal Intelligence, Inner Mongolia University（内蒙古大学时空智能研究中心）；Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University（北航宇航学院航空航天智能科学与技术系）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北航虚拟现实技术与系统国家重点实验室）；China Mobile Communications Group Inner Mongolia Co., Ltd.（中国移动通信集团内蒙古有限公司）
HIMEC: Directional Change Representation and Fixed-Interface Decoding for Remote Sensing Image Change Captioning
University of Electronic Science and Technology of China（电子科技大学）；University of Bonn（波恩大学）；Bonn-Aachen International Center for Information Technology (B-IT)（波恩-亚琛信息技术国际中心（B-IT））
Feature means countenance, remote sensing scene objects with similar characteristics, associated to interesting scene elements in the image formation process. They are classified into three types in image processing, that is low, middle and high. Low level features are color, texture and middle level feature is shape and high level feature is semantic gap of objects. An image retrieval system is a computer system for browsing, searching and retrieving images from a large image database. Content Based Image Retrieval is a technique which uses visual features of image such as color, shape, textu…
INESC-ID；Instituto Superior Técnico（高等技术学院）；University of Lisbon（里斯本大学）；University of Copenhagen（哥本哈根大学）
Inspired by retrieval-augmented language generation and pretrained Vision and Language (V&L) encoders, we present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore, as opposed to the image alone. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT, while the decoder attends to the multimodal encoder representations, benefiting from the extra textual evidence from the retrieved captions. Experimental results on the COCO dataset show that image captioning can be ef…
Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis
Beihang University（北京航空航天大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Astronautics, Beihang University（北京航空航天大学宇航学院）；Shen Yuan Honors College of Beihang University（北京航空航天大学沈元荣誉学院）
HyperCap: Hyperspectral Land Cover Captioning Dataset for Vision Language Models
Department of Computer Science and Engineering, Vellore Institute of Technology（计算机科学与工程系，维洛雷理工学院）；Department of Information Technology, Siddhartha Academy of Higher Education（信息技术系，斯里达拉塔高等教育学院）；Department of Computer Science and Engineering, Indian Institute of Technology, Roorkee（计算机科学与工程系，印度理工学院罗尔基分校）；Department of Computer Science and Engineering, Indraprastha Institute of Information Technology Delhi（计算机科学与工程系，印度信息技术学院德里）；Department of Computer Science and Engineering, Indian Institute of Technology, Kanpur（计算机科学与工程系，印度理工学院坎浦尔）；Department of Computer Science and Artificial Intelligence, University of Granada（计算机科学与人工智能系，格拉纳达大学）；Hyperspectral Computing Laboratory, Department of Computers and Communications, University of Extremadura（超光谱计算实验室，计算机与通信系，埃斯特拉达大学）
Towards Comprehensive Interactive Change Understanding in Remote Sensing: A Large-scale Dataset and Dual-granularity Enhanced VLM
Research Center for Space Computing System, Zhejiang Lab（浙江省实验室空间计算系统研究中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Cyber Science and Engineering, Zhengzhou University（郑州大学网络科学与工程学院）；Liaoning University of Technology（辽宁科技学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University（武汉测绘遥感与信息工程国家重点实验室（LIESMARS））；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering
China Academy of Electronics and Information Technology（中国电子科技集团公司电子科学研究院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology（哈尔滨工业大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）
Mask Approximation Net: A Novel Diffusion Model Approach for Remote Sensing Change Captioning
School of Computer Science and Technology and the Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University（计算机科学与技术学院和教育部智能网络与网络安全重点实验室，西安交通大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（航天信息研究所，中国科学院）；Space Engineering University（航天工程大学）；School of Mathematics and Statistics, Guangdong University of Technology（数学与统计学院，广东工业大学）；Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿-罗斯托克亥姆霍兹中心）
DescribeEarth: Describe Anything for Remote Sensing Images
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；College of Artificial Intelligence, Xi’an Jiaotong University（西安交通大学人工智能学院）；School of Computer Science and Technology and Ministry of Education Key Lab For Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院和教育部智能网络与网络安全重点实验室）；School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学数学与统计学院和教育部智能网络与网络安全重点实验室）；Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China（琶洲实验室（黄埔），广州，广东，中国）
Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model
School of Cyber Science and Technology, University of Science and Technology of China（信息科学技术学院，中国科学技术大学）；China Academy of Electronics and Information Technology（电子信息技术研究院）
