ABC：利用视觉语言模型实现对多模态嵌入的更优控制
ABC: Achieving Better Control of Multimodal Embeddings using VLMs
AI总结：
ABC提出基于视觉语言模型骨干的多模态嵌入模型，通过深度融合图像特征与自然语言指令实现对视觉表示的可控性，并在检索、分类和VQA任务上取得领先性能。
摘要：
视觉嵌入模型在视觉检索和分类等零样本任务中表现出色。然而，这些模型无法用于包含歧义或需要用户指令的任务。这些任务需要一种嵌入模型，其输出能够利用自然语言指令来控制视觉嵌入的表示。现有的基于CLIP的方法独立地嵌入图像和文本，然后融合结果。我们发现这导致模态之间的交互较弱，并且用户对表示的控制能力较差。我们引入了ABC，一个开源的多模态嵌入模型，它使用视觉语言模型骨干来深度整合图像特征与自然语言指令。ABC在MSCOCO图像到文本检索上实现了同规模最佳性能，并且在Massive Multimodal Embedding Benchmark中的分类和VQA任务上是表现最优的模型。凭借高度统一的视觉语言表示，ABC能够使用自然语言解决微妙且可能具有歧义的视觉检索问题。为了评估这一能力，我们设计了CtrlBench，一个需要将文本指令与图像内容交错以进行正确检索的基准。ABC推进了视觉嵌入的最新技术水平，输出具有自然语言控制能力的高质量视觉表示。我们的模型和数据集可在我们的项目页面获取：https://tiger-ai-lab.github.io/ABC/
英文摘要：
Visual embedding models excel at zero-shot tasks like visual retrieval and classification. However, these models cannot be used for tasks that contain ambiguity or require user instruction. These tasks necessitate an embedding model which outputs can use a natural language instruction to control the representation of a visual embedding. Existing CLIP-based approaches embed images and text independently, and fuse the result. We find that this results in weak interactions between modalities, and poor user control over the representation. We introduce ABC, an open-source multimodal embedding model that uses a vision-language model backbone to deeply integrate image features with natural language instructions. ABC achieves best-for-size performance on MSCOCO image-to-text retrieval and is the top performing model on classification and VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly unified vision-language representation, ABC can use natural language to solve subtle and potentially ambiguous visual retrieval problems. To evaluate this capability, we design CtrlBench, a benchmark that requires interleaving textual instructions with image content for correct retrieval. ABC advances the state of visual embeddings, outputting high-quality visual representations with natural language control. Our model and datasets are available at our project page: https://tiger-ai-lab.github.io/ABC/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
