VIRTUE: 视觉-交互文本-图像通用嵌入器
VIRTUE: Visual-Interactive Text-Image Universal Embedder
AI总结：
VIRTUE通过引入视觉交互能力，提升图像和文本的联合表示学习，实现更精确的实体级信息处理和应用拓展。
摘要：
多模态表示学习模型已在复杂任务中展现出成功的应用，而将视觉语言模型（VLMs）整合进来进一步使嵌入模型具备了遵循指令的能力。然而，现有的嵌入模型缺乏视觉交互能力，无法从用户处指定感兴趣区域（例如，点、边界框、掩码），这些能力已在生成模型中被探索以扩大其人机交互的应用范围。给嵌入模型配备视觉交互不仅能够解锁新的应用，使用户意图的局部定位得以实现，这仍是一个未探索的领域，而且还能使模型在图像中学习实体级信息，以补充其对传统嵌入任务的全局表示。在本文中，我们提出了一种新颖的视觉-交互文本-图像通用嵌入器（VIRTUE），将分割模型和视觉语言模型的能力扩展到表示学习领域。在VIRTUE中，分割模型可以处理视觉提示，以定位图像中的特定区域，从而使嵌入器能够更精确地处理复杂和模糊的场景。为了评估VIRTUE的视觉交互能力，我们引入了一个大规模的分割与场景描述检索（SCaR）基准，包含100万条样本，旨在通过同时考虑特定对象和图像场景的实体来检索文本描述。VIRTUE在36个通用MMEB（3.1%-8.5%）和五个视觉交互SCaR（15.2%-20.3%）任务上均实现了最先进的性能。
英文摘要：
Multimodal representation learning models have demonstrated successful operation across complex tasks, and the integration of vision-language models (VLMs) has further enabled embedding models with instruction-following capabilities. However, existing embedding models lack visual-interactive capabilities to specify regions of interest from users (e.g., point, bounding box, mask), which have been explored in generative models to broaden their human-interactive applicability. Equipping embedding models with visual interactions not only would unlock new applications with localized grounding of user intent, which remains unexplored, but also enable the models to learn entity-level information within images to complement their global representations for conventional embedding tasks. In this paper, we propose a novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends the capabilities of the segmentation model and the vision-language model to the realm of representation learning. In VIRTUE, the segmentation model can process visual prompts that pinpoint specific regions within an image, thereby enabling the embedder to handle complex and ambiguous scenarios more precisely. To evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples that aims to retrieve the text caption by jointly considering the entity with a specific object and image scene. VIRTUE consistently achieves a state-of-the-art performance with significant improvements across 36 universal MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
