arXiv 2505.167562025-05-22cs.CVcs.IRcs.MM

表示差异桥接方法用于遥感图像-文本检索

Representation Discrepancy Bridging Method for Remote Sensing Image-Text Retrieval

School of Computer Science and Technology, Xi’an University of Posts and Telecommunications（西安邮电大学计算机科学与技术学院）；Shaanxi Key Laboratory of Network Data Analysis and Intelligent Processing（陕西省网络数据分析与智能处理重点实验室）；Xi’an Key Laboratory of Big Data and Intelligent Computing（西安大数据与智能计算重点实验室）；School of Computer Engineering, Weifang University（潍坊大学计算机工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Department of Electronic and Electrical Engineering, Brunel University of London（伦敦布鲁内尔大学电子与电气工程系）；Dept. Matemàtiques i Informàtica and Institute of Neuroscience, Univeritat de Barcelona（数学与计算机系和神经科学研究所，巴塞罗那大学）

Hailong Ning, Siying Wang, Tao Lei, Xiaopeng Cao, Huanmin Dou, Bin Zhao, Asoke K. Nandi, Petia Radeva

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对遥感图像-文本检索中跨模态优化不平衡问题，提出表示差异桥接方法，通过非对称适配器和双任务一致性损失，在RSICD和RSITMD数据集上显著提升检索性能。

摘要：

遥感图像-文本检索（RSITR）在地理信息解译、灾害监测和城市规划中发挥着关键作用，它通过建立图像与文本描述之间的语义关联来实现上述功能。现有的用于视觉-语言预训练（VLP）模型的参数高效微调（PEFT）方法通常采用对称适配器结构来探索跨模态相关性。然而，文本模态的强判别性可能会主导优化过程，从而抑制图像表示学习。不可忽视的跨模态优化不平衡仍然是提升模型性能的瓶颈。为解决这一问题，本研究提出了一种用于RSITR任务的表示差异桥接（RDB）方法。一方面，设计了一种跨模态非对称适配器（CMAA），以实现模态特定的优化并改善特征对齐。CMAA由视觉增强适配器（VEA）和文本语义适配器（TSA）组成。VEA通过差分注意力（DA）机制挖掘细粒度图像特征，而TSA通过层次注意力（HA）机制识别关键文本语义。另一方面，本研究将传统的单任务检索框架扩展为双任务优化框架，并开发了一种双任务一致性损失（DTCL）。DTCL通过跨模态、分类和指数移动平均一致性约束的自适应加权组合，提高了跨模态对齐的鲁棒性。在RSICD和RSITMD数据集上的实验表明，与最先进的PEFT方法相比，所提出的RDB方法在mR指标上实现了6%-11%的提升，与完全微调的GeoRSCLIP模型相比提升了1.15%-2%。

英文摘要：

Remote Sensing Image-Text Retrieval (RSITR) plays a critical role in geographic information interpretation, disaster monitoring, and urban planning by establishing semantic associations between image and textual descriptions. Existing Parameter-Efficient Fine-Tuning (PEFT) methods for Vision-and-Language Pre-training (VLP) models typically adopt symmetric adapter structures for exploring cross-modal correlations. However, the strong discriminative nature of text modality may dominate the optimization process and inhibits image representation learning. The nonnegligible imbalanced cross-modal optimization remains a bottleneck to enhancing the model performance. To address this issue, this study proposes a Representation Discrepancy Bridging (RDB) method for the RSITR task. On the one hand, a Cross-Modal Asymmetric Adapter (CMAA) is designed to enable modality-specific optimization and improve feature alignment. The CMAA comprises a Visual Enhancement Adapter (VEA) and a Text Semantic Adapter (TSA). VEA mines fine-grained image features by Differential Attention (DA) mechanism, while TSA identifies key textual semantics through Hierarchical Attention (HA) mechanism. On the other hand, this study extends the traditional single-task retrieval framework to a dual-task optimization framework and develops a Dual-Task Consistency Loss (DTCL). The DTCL improves cross-modal alignment robustness through an adaptive weighted combination of cross-modal, classification, and exponential moving average consistency constraints. Experiments on RSICD and RSITMD datasets show that the proposed RDB method achieves a 6%-11% improvement in mR metrics compared to state-of-the-art PEFT methods and a 1.15%-2% improvement over the full fine-tuned GeoRSCLIP model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。