MaskInversion: 通过可解释性图的优化生成局部嵌入
MaskInversion: Localized Embeddings via Optimization of Explainability Maps
AI总结：
MaskInversion通过优化可解释性图生成特定图像区域的嵌入，适用于多种视觉-语言任务。
摘要：
视觉-语言基础模型如CLIP在全局视觉-语言对齐方面取得了巨大成果，但在生成特定图像区域的表示时仍存在一些局限性。为了解决这个问题，我们提出了MaskInversion，一种利用预训练基础模型的特征表示，如CLIP，在测试时为由掩码指定的查询图像区域生成上下文感知嵌入的方法。MaskInversion从初始化嵌入标记开始，并将其可解释性图（从基础模型中导出）与查询掩码进行比较。随后，嵌入标记会进一步细化，以通过最小化其可解释性图与查询掩码之间的差异来近似查询区域。在此过程中，仅更新嵌入向量，而底层基础模型保持冻结，允许MaskInversion与任何预训练模型结合使用。由于导出可解释性图涉及计算梯度，这可能很昂贵，我们提出了一种梯度分解策略来简化此计算。所学习的区域表示可以用于多种任务，包括开放词汇类检索、指代表达理解，以及局部标题生成和图像生成。我们在多个数据集上评估了所提出的方法，如PascalVOC、MSCOCO、RefCOCO和OpenImagesV7，并展示了其与其他最先进的方法相比的能力。
英文摘要：
Vision-language foundation models such as CLIP have achieved tremendous results in global vision-language alignment, but still show some limitations in creating representations for specific image regions. % To address this problem, we propose MaskInversion, a method that leverages the feature representations of pre-trained foundation models, such as CLIP, to generate a context-aware embedding for a query image region specified by a mask at test time. MaskInversion starts with initializing an embedding token and compares its explainability map, derived from the foundation model, to the query mask. The embedding token is then subsequently refined to approximate the query region by minimizing the discrepancy between its explainability map and the query mask. During this process, only the embedding vector is updated, while the underlying foundation model is kept frozen allowing to use MaskInversion with any pre-trained model. As deriving the explainability map involves computing its gradient, which can be expensive, we propose a gradient decomposition strategy that simplifies this computation. The learned region representation can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation. We evaluate the proposed method on all those tasks on several datasets such as PascalVOC, MSCOCO, RefCOCO, and OpenImagesV7 and show its capabilities compared to other SOTA approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
