草图与文本协同：融合结构轮廓和描述属性用于细粒度图像检索
Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval
AI总结：
本文提出STBIR框架，通过融合草图的结构轮廓与文本的色彩纹理信息，提升细粒度图像检索性能，采用课程学习、特征空间优化和多阶段跨模态对齐机制，实验验证其优于现有方法。
摘要：
通过手绘草图或文本描述进行细粒度图像检索仍面临重大挑战，因存在固有模态差距。尽管手绘草图能捕捉复杂的结构轮廓，但缺乏颜色和纹理信息，而文本虽能提供这些信息却缺失空间轮廓。受这两种模态互补性的启发，我们提出了Sketch and Text Based Image Retrieval (STBIR)框架。通过将文本丰富的颜色和纹理线索与草图提供的结构轮廓相结合，STBIR实现了优越的细粒度检索性能。首先，我们提出了一种由课程学习驱动的鲁棒性增强模块，以增强模型在处理不同质量查询时的鲁棒性。其次，我们引入了基于类别知识的特征空间优化模块，从而显著提升模型的表征能力。最后，我们设计了多阶段跨模态特征对齐机制，以有效缓解跨模态特征对齐的挑战。此外，我们整理了细粒度STBIR基准数据集，以严格验证所提框架的有效性，并为后续相关研究提供数据支持。大量实验表明，所提出的STBIR框架在性能上显著优于现有最先进方法。
英文摘要：
Fine-grained image retrieval via hand-drawn sketches or textual descriptions remains a critical challenge due to inherent modality gaps. While hand-drawn sketches capture complex structural contours, they lack color and texture, which text effectively provides despite omitting spatial contours. Motivated by the complementary nature of these modalities, we propose the Sketch and Text Based Image Retrieval (STBIR) framework. By synergizing the rich color and texture cues from text with the structural outlines provided by sketches, STBIR achieves superior fine-grained retrieval performance. First, a curriculum learning driven robustness enhancement module is proposed to enhance the model's robustness when handling queries of varying quality. Second, we introduce a category-knowledge-based feature space optimization module, thereby significantly boosting the model's representational power. Finally, we design a multi-stage cross-modal feature alignment mechanism to effectively mitigate the challenges of cross modal feature alignment. Furthermore, we curate the fine-grained STBIR benchmark dataset to rigorously validate the efficacy of our proposed framework and to provide data support as a reference for subsequent related research. Extensive experiments demonstrate that the proposed STBIR framework significantly outperforms state of the art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
