LMM正则化的CLIP嵌入用于图像分类
LMM-Regularized CLIP Embeddings for Image Classification
AI总结：
本文提出一种基于大型多模态模型的正则化方法，利用LMM生成语义描述来约束CLIP图像嵌入，增强判别能力，提升图像分类性能，并在三个数据集上验证了有效性。
摘要：
在本文中，我们利用强大的CLIP视觉-语言模型处理图像分类任务。我们的目标是通过提出一种新颖的基于大型多模态模型（LMM）的正则化方法，利用CLIP的图像编码器来提升分类性能。所提出的方法使用LMM为数据集中的图像提取语义描述。然后，它使用冻结的CLIP文本编码器来获取相应的文本嵌入，并计算平均语义类描述。随后，我们通过添加一个分类头来调整CLIP的图像编码器，并除了主分类目标外，还使用一个额外的辅助目标对图像编码器输出进行训练。该额外目标迫使图像编码器输出处的嵌入与其对应的LMM生成的平均语义类描述变得相似。通过这种方式，它产生了具有增强判别能力的嵌入，从而提高了分类性能。所提出的正则化方法的有效性通过在三个图像分类数据集上的大量实验得到了验证。
英文摘要：
In this paper we deal with image classification tasks using the powerful CLIP vision-language model. Our goal is to advance the classification performance using the CLIP's image encoder, by proposing a novel Large Multimodal Model (LMM) based regularization method. The proposed method uses an LMM to extract semantic descriptions for the images of the dataset. Then, it uses the CLIP's text encoder, frozen, in order to obtain the corresponding text embeddings and compute the mean semantic class descriptions. Subsequently, we adapt the CLIP's image encoder by adding a classification head, and we train it along with the image encoder output, apart from the main classification objective, with an additional auxiliary objective. The additional objective forces the embeddings at the image encoder's output to become similar to their corresponding LMM-generated mean semantic class descriptions. In this way, it produces embeddings with enhanced discrimination ability, leading to improved classification performance. The effectiveness of the proposed regularization method is validated through extensive experiments on three image classification datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
