arXiv 2607.061852026-07-08cs.CV

用于细粒度图像识别的视觉语言模型中的结构化压缩提示调优

Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition

School of Information Science and Technology, Northwest University（西北大学信息科学与技术学院）；Laboratory of Intelligent Information Processing, Institute of Computing Technology（中国科学院计算技术研究所智能信息处理重点实验室）

Xinda Liu, Qinyu Zhang, Weiqing Min, Guohua Geng, Shuqiang Jiang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对细粒度图像识别中视觉语言模型捕捉细微差别能力有限的问题，提出结构化压缩提示调优（SCPT），通过语义关系编码和语义压缩损失增强语义结构建模，在多基准实验中有效减轻语义模糊性，取得领先性能。

摘要：

细粒度图像识别因手动标注需要大量专业知识和精力而面临重大挑战。像CLIP这样的视觉语言模型提供了一种有吸引力的零样本替代方案，减少了对大量标注数据的依赖。然而，它们捕捉细微差别的能力仍然有限，导致识别性能不佳。虽然提示调优已被证明对调整视觉语言模型有效，但大多数现有方法将类别标签视为孤立的离散实体，忽略了它们之间丰富的语义关系。这种过度简化的假设限制了模型捕捉层次依赖性和类间相关性的能力，而这对于区分视觉上相似的类别至关重要。在细粒度分类中，这个问题尤为严重，因为准确的识别依赖于理解复杂的标签语义。为了解决这个问题，我们提出了结构化压缩提示调优（SCPT），它增强了提示学习中的语义结构建模。具体来说，我们引入了语义关系编码（SRE）来明确地对类间语义拓扑进行建模，并编码结构化的标签关系。同时，我们设计了一种语义压缩损失（ScLoss）来抑制冗余监督，并从全局语义空间中提取有区分力的成分。这些组件一起显著提高了语义对齐和细粒度区分能力。在14个细粒度基准上进行的广泛实验表明，SCPT有效地减轻了语义模糊性，并在少样本和从基础到新类别泛化设置中都取得了领先的性能。

英文摘要：

Fine-grained image recognition poses a significant challenge due to the substantial expertise and effort required for manual annotation. Vision-language models (VLMs) like CLIP provide a compelling zero-shot alternative, reducing reliance on extensive labeled data. However, their ability to capture subtle distinctions remains limited, leading to subpar recognition performance. While prompt tuning has proven effective for adapting VLMs, most existing methods treat class labels as isolated, discrete entities, overlooking the rich semantic relationships between them. This oversimplified assumption limits the model's ability to capture hierarchical dependencies and inter-class correlations -- both critical for distinguishing visually similar categories. The problem is especially acute in fine-grained classification, where accurate recognition depends on understanding complex label semantics. To address this, we propose Structured-Condensed Prompt Tuning (SCPT), which enhances semantic structure modeling in prompt learning. Specifically, we introduce Semantic Relation Encoding (SRE) to explicitly model inter-class semantic topology and encode structured label relationships. In parallel, we design a Semantic Condensation loss (ScLoss) to suppress redundant supervision and extract discriminative components from the global semantic space. Together, these components significantly improve semantic alignment and fine-grained discrimination. Extensive experiments on 14 fine-grained benchmarks show that SCPT effectively mitigates semantic ambiguity and achieves state-of-the-art performance in both few-shot and base-to-novel generalization settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。