通过文本描述适配视觉语言模型以改进零样本分类
Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions
AI总结：
本文提出利用 LLM 生成的类别描述与细粒度图像数据，以袋级图文监督训练并适配 VLM，从而提升新类别零样本分类性能。
摘要：
现有视觉语言模型（VLM，如 CLIP）的零样本性能受到特定领域大规模对齐图像与文本数据集可得性的限制。在本工作中，我们利用两类互补信息来源——由大型语言模型（LLM）生成的类别描述，以及丰富的细粒度图像分类数据集——来提升 VLM 在各细粒度领域中的零样本分类性能。在技术方面，我们开发了利用这种“袋级”图文监督训练 VLM 的方法。我们发现，在测试时简单使用这些属性并不能提升性能，但我们的训练策略（例如在 iNaturalist 数据集上）使鸟类和花卉等新类别的零样本分类准确率平均提高 4–5%。在使用类别子集对模型进行微调的领域中，也观察到类似提升。通过以多种方式提示 LLM，我们生成涵盖视觉外观、栖息地和地理区域的描述，并将其与类别分类结构等现有属性配对。我们系统评估了这些描述提升自然领域零样本分类的能力。研究结果表明，地理先验可以与视觉外观同样有效，且二者互补。我们的方法还优于此前基于提示对 VLM 进行调优的工作。我们发布了由 14 个数据集组成的基准，地址为 https://github.com/cvl-umass/AdaptCLIPZS ，这将有助于未来的零样本识别研究。
英文摘要：
The zero-shot performance of existing vision-language models (VLMs) such as CLIP is limited by the availability of large-scale, aligned image and text datasets in specific domains. In this work, we leverage two complementary sources of information -- descriptions of categories generated by large language models (LLMs) and abundant, fine-grained image classification datasets -- to improve the zero-shot classification performance of VLMs across fine-grained domains. On the technical side, we develop methods to train VLMs with this "bag-level" image-text supervision. We find that simply using these attributes at test-time does not improve performance, but our training strategy, for example, on the iNaturalist dataset, leads to an average improvement of 4-5% in zero-shot classification accuracy for novel categories of birds and flowers. Similar improvements are observed in domains where a subset of the categories was used to fine-tune the model. By prompting LLMs in various ways, we generate descriptions that capture visual appearance, habitat, and geographic regions and pair them with existing attributes such as the taxonomic structure of the categories. We systematically evaluate their ability to improve zero-shot categorization in natural domains. Our findings suggest that geographic priors can be just as effective and are complementary to visual appearance. Our method also outperforms prior work on prompt-based tuning of VLMs. We release the benchmark, consisting of 14 datasets at https://github.com/cvl-umass/AdaptCLIPZS , which will contribute to future research in zero-shot recognition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
