基于多层次注意力引导分词的零样本草图遥感图像检索
Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenization
AI总结：
针对现有草图遥感图像检索中多层次特征挖掘不足的问题，提出融合多层次特征提取、注意力引导分词与跨模态注意力更新的零样本检索方法，性能显著优于现有技术，泛化性强且可通过预计算提升扩展性，代码与数据集已公开。
摘要：
在遥感大数据领域，从遥感数据库中有效且高效地检索图像是一项关键挑战。使用手绘草图作为检索输入具有直观、用户友好的优势，但草图的多层次特征整合潜力尚未得到充分挖掘，导致检索性能欠佳。为弥补这一不足，本研究提出了一种新型零样本草图遥感图像检索方法，该方法融合了多层次特征提取、自注意力引导的分词与过滤以及跨模态注意力更新机制。该方法仅利用视觉信息，无需关于草图与图像的语义知识。其首先通过多层次自注意力引导的特征提取对查询草图进行分词，同时利用自注意力特征提取对候选图像进行分词；随后采用交叉注意力机制建立两种模态之间的分词对应关系，以辅助计算草图与图像的相似度。在多个数据集上的测试表明，本方法的性能显著优于现有的草图遥感图像检索技术。值得注意的是，该方法在处理未见过的类别和新型遥感数据时，还展现出强大的零样本学习能力和良好的泛化性。通过预计算数据库中所有候选图像的检索分词，可进一步提升该方法的可扩展性。本研究凸显了多层次注意力引导分词在跨模态遥感图像检索中的巨大潜力。为便于更广泛的获取与研究推进，我们已将本研究使用的代码和数据集在线公开，代码与数据集可访问https://github.com/Snowstormfly/Cross-modal-retrieval-MLAGT获取。
英文摘要：
Effectively and efficiently retrieving images from remote sensing databases is a critical challenge in the realm of remote sensing big data. Utilizing hand-drawn sketches as retrieval inputs offers intuitive and user-friendly advantages, yet the potential of multi-level feature integration from sketches remains underexplored, leading to suboptimal retrieval performance. To address this gap, our study introduces a novel zero-shot, sketch-based retrieval method for remote sensing images, leveraging multi-level feature extraction, self-attention-guided tokenization and filtering, and cross-modality attention update. This approach employs only vision information and does not require semantic knowledge concerning the sketch and image. It starts by employing multi-level self-attention guided feature extraction to tokenize the query sketches, as well as self-attention feature extraction to tokenize the candidate images. It then employs cross-attention mechanisms to establish token correspondence between these two modalities, facilitating the computation of sketch-to-image similarity. Our method significantly outperforms existing sketch-based remote sensing image retrieval techniques, as evidenced by tests on multiple datasets. Notably, it also exhibits robust zero-shot learning capabilities and strong generalizability in handling unseen categories and novel remote sensing data. The method's scalability can be further enhanced by the pre-calculation of retrieval tokens for all candidate images in a database. This research underscores the significant potential of multi-level, attention-guided tokenization in cross-modal remote sensing image retrieval. For broader accessibility and research facilitation, we have made the code and dataset used in this study publicly available online. Code and dataset are available at https://github.com/Snowstormfly/Cross-modal-retrieval-MLAGT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
