迈向真实零样本分类：通过自结构语义对齐
Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment
AI总结：
针对无标注但词汇广泛的真实零样本分类，提出S^3A框架，通过CVPR算法从无标注数据中提取结构语义并自我学习，显著提升CLIP分类准确率。
摘要：
大规模预训练的视觉语言模型（VLMs）已被证明在零样本分类中有效。尽管取得了成功，大多数基于VLMs的传统方法受限于部分源监督或理想词汇表的假设，这很难满足开放世界的场景。在本文中，我们针对一个更具挑战性的设置，即真实零样本分类，该设置假设没有标注，而是有一个广泛的词汇表。为了解决这一挑战，我们提出了自结构语义对齐（S^3A）框架，该框架从无标注数据中提取结构语义信息，同时进行自我学习。我们的S^3A框架采用了一种独特的聚类-投票-提示-重对齐（CVPR）算法，该算法迭代地对无标注数据进行分组，以推导出用于伪监督的结构语义。我们的CVPR过程包括对图像进行迭代聚类，在每个聚类内进行投票以从词汇表中识别初始类别候选，使用大型语言模型生成判别性提示以区分混淆的候选，并将图像和词汇表重新对齐作为结构语义对齐。最后，我们提出通过教师-学生学习策略，同时使用个体和结构语义对齐来自我学习CLIP图像编码器。我们在各种通用和细粒度基准上的全面实验表明，S^3A方法相对于现有的基于VLMs的方法提供了显著的改进，平均比CLIP提高了超过15%的准确率。我们的代码、模型和提示已在https://github.com/sheng-eatamath/S3A公开。
英文摘要：
Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. Despite the success, most traditional VLMs-based methods are restricted by the assumption of partial source supervision or ideal vocabularies, which rarely satisfy the open-world scenario. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. To address this challenge, we propose the Self Structural Semantic Alignment (S^3A) framework, which extracts the structural semantic information from unlabeled data while simultaneously self-learning. Our S^3A framework adopts a unique Cluster-Vote-Prompt-Realign (CVPR) algorithm, which iteratively groups unlabeled data to derive structural semantics for pseudo-supervision. Our CVPR process includes iterative clustering on images, voting within each cluster to identify initial class candidates from the vocabulary, generating discriminative prompts with large language models to discern confusing candidates, and realigning images and the vocabulary as structural semantic alignment. Finally, we propose to self-learn the CLIP image encoder with both individual and structural semantic alignment through a teacher-student learning strategy. Our comprehensive experiments across various generic and fine-grained benchmarks demonstrate that the S^3A method offers substantial improvements over existing VLMs-based approaches, achieving a more than 15% accuracy improvement over CLIP on average. Our codes, models, and prompts are publicly released at https://github.com/sheng-eatamath/S3A.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
