MOFI：从含噪实体标注图像中学习图像表示
MOFI: Learning Image Representations from Noisy Entity Annotated Images
AI总结：
MOFI是一种视觉基础模型，通过从含噪图像-文本对自动生成实体标签的I2E数据集进行预训练，在图像检索和分类任务上超越了CLIP，展示了细粒度实体监督的有效性。
摘要：
我们提出了MOFI（Manifold OF Images），一种新的视觉基础模型，旨在从含噪实体标注图像中学习图像表示。MOFI与先前工作的不同之处在于两个关键方面：（i）预训练数据和（ii）训练方案。关于数据，我们引入了一种新方法，从含噪图像-文本对中自动为图像分配实体标签。我们的方法包括使用命名实体识别模型从替代文本中提取实体，然后使用CLIP模型选择正确的实体作为配对图像的标签。这是一种简单、成本效益高的方法，可以扩展到处理数十亿个网络挖掘的图像-文本对。通过这种方法，我们创建了Image-to-Entities（I2E）数据集，包含10亿张图像和200万个不同实体，覆盖了野外丰富的视觉概念。基于I2E数据集，我们研究了不同的训练方案，如监督预训练、对比预训练和多任务学习。对于对比预训练，我们将实体名称视为自由形式的文本，并进一步用实体描述丰富它们。实验表明，使用大规模细粒度实体标签进行监督预训练对图像检索任务非常有效，而多任务训练进一步提高了性能。最终的MOFI模型在具有挑战性的GPR1200数据集上达到了86.66%的mAP，超过了OpenAI的CLIP模型之前最先进的72.19%的性能。在零样本和线性探针图像分类上的进一步实验也表明，MOFI优于在原始图像-文本数据上训练的CLIP模型，证明了I2E数据集在学习强图像表示方面的有效性。我们在https://github.com/apple/ml-mofi上发布了我们的代码和模型权重。
英文摘要：
We present MOFI, Manifold OF Images, a new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: (i) pre-training data, and (ii) training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image. It's a simple, cost-effective method that can scale to handle billions of web-mined image-text pairs. Through this method, we have created Image-to-Entities (I2E), a new dataset with 1 billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the I2E dataset, we study different training recipes like supervised pre-training, contrastive pre-training, and multi-task learning. For contrastive pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66% mAP on the challenging GPR1200 dataset, surpassing the previous state-of-the-art performance of 72.19% from OpenAI's CLIP model. Further experiments on zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the I2E dataset in learning strong image representations. We release our code and model weights at https://github.com/apple/ml-mofi.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
