跨域微调中注意原型与图像之间的差距
Mind the Gap Between Prototypes and Images in Cross-domain Finetuning
AI总结：
针对跨域少样本分类中原型与图像嵌入共享同一变换会限制表示学习的问题，本文提出对比式原型-图像适配方法 CoPA，分别适配原型和图像变换，在 Meta-Dataset 上以更高效方式取得最优性能。
摘要：
在跨域少样本分类（CFC）中，近期工作主要关注在冻结的预训练骨干网络之上，利用少量标注数据适配一个简单的变换头，将嵌入投影到任务特定的度量空间，在该空间中通过度量图像实例与原型表示之间的相似度来完成分类。从技术上讲，此类框架隐含采用的一个假设是原型嵌入和图像实例嵌入共享相同的表示变换。然而，在本文中，我们发现从冻结的预训练骨干网络中提取的原型嵌入与图像实例嵌入之间天然存在一个差距，该差距类似于模态差距，而在适配阶段简单地应用相同的变换会限制对最优表示的探索，并缩小原型与图像表示之间的差距。为了解决这一问题，我们提出了一种简单而有效的方法，即对比式原型-图像适配（CoPA），该方法类似于 CLIP，通过将原型视为文本提示，分别对原型和图像适配不同的变换。在 Meta-Dataset 上的大量实验表明，CoPA 以更高效的方式取得了最先进的性能。同时，进一步的分析还表明，CoPA 能够学习到更好的表示簇、扩大该差距，并在差距扩大处取得最小的验证损失。
英文摘要：
In cross-domain few-shot classification (CFC), recent works mainly focus on adapting a simple transformation head on top of a frozen pre-trained backbone with few labeled data to project embeddings into a task-specific metric space where classification can be performed by measuring similarities between image instance and prototype representations. Technically, an assumption implicitly adopted in such a framework is that the prototype and image instance embeddings share the same representation transformation. However, in this paper, we find that there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings extracted from the frozen pre-trained backbone, and simply applying the same transformation during the adaptation phase constrains exploring the optimal representations and shrinks the gap between prototype and image representations. To solve this problem, we propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations respectively for prototypes and images similarly to CLIP by treating prototypes as text prompts. Extensive experiments on Meta-Dataset demonstrate that CoPA achieves the state-of-the-art performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve minimal validation loss at the enlarged gap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
