生成、转导、适应：基于视觉语言模型的迭代转导
Generate, Transduct, Adapt: Iterative Transduction with VLMs
AI总结：
提出GTA-CLIP，通过迭代生成属性、转导推理和编码器微调，在语言与视觉空间联合转导，零样本下平均提升8.6%和3.7%。
摘要：
基于视觉语言模型的转导式零样本学习利用数据集内的图像-图像相似性，相较于归纳式设置，能够实现更高的分类准确率。然而，在此背景下，探索语言空间结构的工作却很少。我们提出了GTA-CLIP，一种新颖的技术，它融合了来自语言模型的监督，用于在语言和视觉空间中进行联合转导。我们的方法是迭代式的，包含三个步骤：（i）通过查询语言模型逐步探索属性空间，（ii）一种属性增强的转导推理过程，以及（iii）基于数据集内推断出的标签对语言和视觉编码器进行微调。通过使用CLIP编码器的实验，我们证明了在零样本设置下，GTA-CLIP在12个数据集和3个编码器上，相较于CLIP和转导式CLIP，平均性能分别提升了8.6%和3.7%。我们还在少样本设置中观察到了类似的改进。我们进行了消融研究，证明了每一步的价值，并可视化了在转导学习驱动下，视觉和语言空间在迭代过程中的演变。代码已发布在https://github.com/cvl-umass/GTA-CLIP。
英文摘要：
Transductive zero-shot learning with vision-language models leverages image-image similarities within the dataset to achieve better classification accuracy compared to the inductive setting. However, there is little work that explores the structure of the language space in this context. We propose GTA-CLIP, a novel technique that incorporates supervision from language models for joint transduction in language and vision spaces. Our approach is iterative and consists of three steps: (i) incrementally exploring the attribute space by querying language models, (ii) an attribute-augmented transductive inference procedure, and (iii) fine-tuning the language and vision encoders based on inferred labels within the dataset. Through experiments with CLIP encoders, we demonstrate that GTA-CLIP, yields an average performance improvement of 8.6% and 3.7% across 12 datasets and 3 encoders, over CLIP and transductive CLIP respectively in the zero-shot setting. We also observe similar improvements in a few-shot setting. We present ablation studies that demonstrate the value of each step and visualize how the vision and language spaces evolve over iterations driven by the transductive learning. Code is released at https://github.com/cvl-umass/GTA-CLIP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
