为视觉和语言模型学习类别的命名
Learning to Name Classes for Vision and Language Models
AI总结：
提出在冻结的视觉语言模型上为每个类学习最优词嵌入，以解决手工类名敏感和难以适应小数据集的问题，保留零样本能力并提升图像分类与目标检测性能。
摘要：
大规模视觉和语言模型通过将特定类别的文本查询映射到图像内容，可以实现令人印象深刻的零样本识别性能。然而，仍然存在两个不同的挑战：对定义查询的手工类名选择高度敏感，以及难以适应新的、较小的数据集。为了解决这些问题，我们提出利用可用数据为每个类学习一个作为视觉内容函数的最优词嵌入。通过在冻结的模型上学习新的词嵌入，我们能够保留新类别的零样本能力，轻松适应新数据集，并调整潜在错误、非描述性或歧义的类名。我们证明该方案可集成到图像分类和目标检测流程中，在多种场景下取得显著性能提升，并为模型偏差和标注错误提供洞察。
英文摘要：
Large scale vision and language models can achieve impressive zero-shot recognition performance by mapping class specific text queries to image content. Two distinct challenges that remain however, are high sensitivity to the choice of handcrafted class names that define queries, and the difficulty of adaptation to new, smaller datasets. Towards addressing these problems, we propose to leverage available data to learn, for each class, an optimal word embedding as a function of the visual content. By learning new word embeddings on an otherwise frozen model, we are able to retain zero-shot capabilities for new classes, easily adapt models to new datasets, and adjust potentially erroneous, non-descriptive or ambiguous class names. We show that our solution can easily be integrated in image classification and object detection pipelines, yields significant performance gains in multiple scenarios and provides insights into model biases and labelling errors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
