HGCLIP：探索利用图表示的视觉-语言模型以实现层次化理解
HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding
AI总结：
提出 HGCLIP 框架，通过图表示学习将类别层次注入 CLIP 的文本与图像特征，在 11 个视觉识别基准上显著提升多粒度层次分类性能。
摘要：
物体类别通常被组织成一个多粒度的分类层次结构。在对不同层次级别的类别进行分类时，传统的单模态方法主要关注图像特征，在复杂场景中暴露出局限性。近期将视觉-语言模型（VLM）与类别层次结构相结合的研究展现出良好前景，但仍未能充分利用层次关系。这些工作受限于无法在不同粒度的类别上有效执行。为解决这一问题，我们提出了一个新颖框架 HGCLIP，它将 CLIP 与通过图表示学习对层次化类别结构进行更深入利用有效结合。我们探索将类别层次结构构建为图，其节点表示每个类别的文本或图像特征。经过图编码器后，文本特征融入层次结构信息，而图像特征则通过注意力机制强调从原型获得的类别感知特征。我们的方法在 11 个多样化的视觉识别基准上展现出显著提升。代码已完全开放于 https://github.com/richard-peng-xia/HGCLIP。
英文摘要：
Object categories are typically organized into a multi-granularity taxonomic hierarchy. When classifying categories at different hierarchy levels, traditional uni-modal approaches focus primarily on image features, revealing limitations in complex scenarios. Recent studies integrating Vision-Language Models (VLMs) with class hierarchies have shown promise, yet they fall short of fully exploiting the hierarchical relationships. These efforts are constrained by their inability to perform effectively across varied granularity of categories. To tackle this issue, we propose a novel framework (HGCLIP) that effectively combines CLIP with a deeper exploitation of the Hierarchical class structure via Graph representation learning. We explore constructing the class hierarchy into a graph, with its nodes representing the textual or image features of each category. After passing through a graph encoder, the textual features incorporate hierarchical structure information, while the image features emphasize class-aware features derived from prototypes through the attention mechanism. Our approach demonstrates significant improvements on 11 diverse visual recognition benchmarks. Our codes are fully available at https://github.com/richard-peng-xia/HGCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
