如果你能描述它，他们就能看到它：从文本描述中跨模态学习视觉概念
If you can describe it, they can see it: Cross-Modal Learning of Visual Concepts from Textual Descriptions
AI总结：
本文提出通过文本描述跨模态学习视觉概念的方法，利用知识迁移技术提升视觉-语言模型的零样本性能。
摘要：
人类可以根据其经验和先前知识，从自然语言描述中可视化新的和未知的概念。受此启发，我们提出了一种方法，将这种能力扩展到视觉-语言模型（VLMs），通过仅使用文本描述来教授它们新的概念。我们称之为知识迁移（KT）。我们的假设是预训练VLM的知识可以重新利用以表示以前未知的概念。在提供新概念的文本描述时，KT通过通过模型反向获得的视觉编码器相关特征，对齐到其文本表示。与依赖视觉示例或外部生成模型的方法不同，KT通过直接从文本注入视觉知识，在同一VLM内转移知识。通过在多个VLM任务上的广泛评估，包括分类、分割、图像-文本检索和描述生成，我们展示了：1）KT可以高效地从单个文本描述中引入新的视觉概念；2）同样的原理可以用于细化现有概念的表示；3）KT显著提高了零样本VLMs的性能。
英文摘要：
Humans can visualize new and unknown concepts from their natural language description, based on their experience and previous knowledge. Insipired by this, we present a way to extend this ability to Vision-Language Models (VLMs), teaching them novel concepts by only using a textual description. We refer to this approach as Knowledge Transfer (KT). Our hypothesis is that the knowledge of a pre-trained VLM can be re-used to represent previously unknown concepts. Provided with a textual description of the novel concept, KT works by aligning relevant features of the visual encoder, obtained through model inversion, to its text representation. Differently from approaches relying on visual examples or external generative models, KT transfers knowledge within the same VLM by injecting visual knowledge directly from the text. Through an extensive evaluation on several VLM tasks, including classification, segmentation, image-text retrieval, and captioning, we show that: 1) KT can efficiently introduce new visual concepts from a single textual description; 2) the same principle can be used to refine the representation of existing concepts; and 3) KT significantly improves the performance of zero-shot VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
