StructXLIP: 通过多模态结构线索增强视觉-语言模型
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
AI总结：
StructXLIP通过引入多模态结构线索提升视觉-语言模型的跨模态检索性能。
摘要：
基于边缘的表示是视觉理解的基本线索，这一原理源自早期视觉研究，至今仍然至关重要。我们将其扩展到视觉-语言对齐中，证明通过隔离和对齐跨模态的结构线索可以显著提升在长且细节丰富的描述词上进行微调的效果，特别是提高跨模态检索。我们介绍了StructXLIP，一种微调对齐范式，它提取边缘图（例如Canny），将其视为图像视觉结构的代理，并过滤相应的描述词以强调结构线索，使其成为“结构导向”的。微调在标准对齐损失的基础上增加了三种结构导向的损失：（i）将边缘图与结构文本对齐，（ii）将局部边缘区域与文本片段匹配，（iii）将边缘图与颜色图像连接以防止表征漂移。从理论角度看，尽管标准CLIP最大化了视觉和文本嵌入之间的互信息，但StructXLIP还最大化了多模态结构表示之间的互信息。这种辅助优化本质上更困难，引导模型向更稳健和语义稳定的最小值发展，增强视觉-语言对齐。除了在通用和专业领域均优于当前竞争对手的跨模态检索外，我们的方法还作为一种通用的提升配方，可以以即插即用的方式集成到未来的方法中。代码和预训练模型已公开在：https://github.com/intelligolabs/StructXLIP。
英文摘要：
Edge-based representations are fundamental cues for visual understanding, a principle rooted in early vision research and still central today. We extend this principle to vision-language alignment, showing that isolating and aligning structural cues across modalities can greatly benefit fine-tuning on long, detail-rich captions, with a specific focus on improving cross-modal retrieval. We introduce StructXLIP, a fine-tuning alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image, and filters the corresponding captions to emphasize structural cues, making them "structure-centric". Fine-tuning augments the standard alignment loss with three structure-centric losses: (i) aligning edge maps with structural text, (ii) matching local edge regions to textual chunks, and (iii) connecting edge maps to color images to prevent representation drift. From a theoretical standpoint, while standard CLIP maximizes the mutual information between visual and textual embeddings, StructXLIP additionally maximizes the mutual information between multimodal structural representations. This auxiliary optimization is intrinsically harder, guiding the model toward more robust and semantically stable minima, enhancing vision-language alignment. Beyond outperforming current competitors on cross-modal retrieval in both general and specialized domains, our method serves as a general boosting recipe that can be integrated into future approaches in a plug-and-play manner. Code and pretrained models are publicly available at: https://github.com/intelligolabs/StructXLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
