HyperCLIP：使用超网络适配视觉-语言模型
HyperCLIP: Adapting Vision-Language models with Hypernetworks
AI总结：
本文提出名为HyperCLIP的视觉-语言架构，通过超网络动态调整小型图像编码器权重以适配文本输入，在极小训练开销下显著提升零样本图像分类准确率。
摘要：
基于对比目标训练的自监督视觉-语言模型构成了当前AI视觉任务中最先进方法的基础。这些模型的成功是使用海量网络规模数据集训练的直接结果，但它们需要相应的大型视觉组件才能从如此广泛的数据域中正确学习强大且通用的表示。这给部署大型视觉-语言模型带来了挑战，尤其是在资源受限的环境中。为了解决这一问题，我们提出了一种替代的视觉-语言架构，称为HyperCLIP，它使用小型图像编码器和一个超网络，该超网络能够针对每组新的文本输入动态调整图像编码器权重。模型的三个组件（超网络、图像编码器和文本编码器）均以端到端的方式进行联合预训练。借助训练好的HyperCLIP模型，我们只需通过文本编码器和超网络进行一次前向传递，即可为任意任务生成新的零样本、部署友好的图像分类器。HyperCLIP在ImageNet上将使用小型图像编码器训练的SigLIP模型的零样本准确率提升了3%，在CIFAR-100上提升了5%，且训练吞吐量开销极小。
英文摘要：
Self-supervised vision-language models trained with contrastive objectives form the basis of current state-of-the-art methods in AI vision tasks. The success of these models is a direct consequence of the huge web-scale datasets used to train them, but they require correspondingly large vision components to properly learn powerful and general representations from such a broad data domain. This poses a challenge for deploying large vision-language models, especially in resource-constrained environments. To address this, we propose an alternate vision-language architecture, called HyperCLIP, that uses a small image encoder along with a hypernetwork that dynamically adapts image encoder weights to each new set of text inputs. All three components of the model (hypernetwork, image encoder, and text encoder) are pre-trained jointly end-to-end, and with a trained HyperCLIP model, we can generate new zero-shot deployment-friendly image classifiers for any task with a single forward pass through the text encoder and hypernetwork. HyperCLIP increases the zero-shot accuracy of SigLIP trained models with small image encoders by up to 3% on ImageNet and 5% on CIFAR-100 with minimal training throughput overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
