JSSFF：一种用于遥感图像描述的联合结构-语义融合框架
JSSFF: A Joint Structural-Semantic Fusion Framework for Remote Sensing Image Captioning
AI总结：
本文提出JSSFF框架，通过融合图像和边缘信息提升遥感图像描述的准确性和语义表达。
摘要：
编码器-解码器框架如今广受青睐。在此模型中，编码器从输入图像中提取信息性视觉特征，解码器采用序列到序列的公式生成对应的文本描述。现有模型更侧重于决策部分。然而，从图像中提取有意义的信息可以帮助解码器生成准确的描述，通过提供关于物体及其关系的信息。遥感图像高度复杂。一个主要挑战是检测超出可见边界的目标，由于遮挡、重叠结构和模糊边缘。因此，需要设计一种能够有效捕捉高层语义和低层空间细节的方法以实现准确的描述生成。在本工作中，我们提出了一种边缘感知融合方法，通过将原始图像及其边缘感知版本纳入编码器中，以增强特征表示和边界意识。我们使用基于比较的束搜索（CBBS）生成描述，通过公平比较候选描述来实现定量指标和定性描述相关性之间的平衡。实验结果表明，我们的模型在定量和定性方面均优于几种基线模型。
英文摘要：
The encoder-decoder framework has become widely popular nowadays. In this model, the encoder extracts informative visual features from an input image, and the decoder employs a sequence-to-sequence formulation to generate the corresponding textual description from these features. The existing models focus more on the decision part. However, extracting meaningful information from the image can help the decoder generate an accurate caption by providing information about the objects and their relationship. Remote sensing images are highly complex. One major challenge is detecting objects that extend beyond their visible boundaries due to occlusion, overlapping structures, and unclear edges. Hence, there is a need to design an approach that can effectively capture both high-level semantics and low-level spatial details for accurate caption generation. In this work, we have proposed an edge-aware fusion method by incorporating the original image and its edge-aware version into the encoder to enhance feature representation and boundary awareness. We used a comparison-based beam search (CBBS) to generate captions to achieve a balanced trade-off between quantitative metrics and qualitative caption relevance through fairness-based comparison of candidate captions. Experimental results demonstrate our model's superiority over several baseline models in quantitative and qualitative perspectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
