arXiv 2507.103182025-07-14cs.CV

Mind the Gap：将视觉基础模型对齐到图像特征匹配

Mind the Gap: Aligning Vision Foundation Models to Image Feature Matching

National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）

Yuhan Liu, Jingwen Fu, Yang Wu, Kangyi Wu, Pengna Li, Jiayi Wu, Sanping Zhou, Jingmin Xin

备注：: Accepted by ICCV 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉基础模型在图像特征匹配中的不对齐问题，提出基于扩散模型的IMD框架，通过捕获实例级细节和跨图像交互提示模块，在多实例场景新基准IMIM上提升12%，达到SOTA。

摘要：

利用视觉基础模型已成为提升图像特征匹配性能的主流范式。然而，以往的研究在将基础模型引入特征匹配时忽略了不对齐问题。这种不对齐源于基础模型专注于单图像理解与特征匹配所需的跨图像理解之间的差异。具体而言：1）常用基础模型生成的嵌入与特征匹配所需的最优嵌入存在差异；2）缺乏将单图像理解能力有效转化为跨图像理解的有效机制。不对齐的一个显著后果是，在处理多实例特征匹配问题时表现不佳。为解决此问题，我们引入了一个简单但有效的框架，称为IMD（基于预训练扩散模型的图像特征匹配），包含两部分：1）与采用强调全局语义的对比学习基础模型的主流方案不同，我们整合了基于生成的扩散模型以有效捕获实例级细节。2）我们利用生成模型中的提示机制作为自然通道，提出了一种新颖的跨图像交互提示模块，以促进图像对之间的双向信息交互。为更准确地衡量不对齐，我们提出了一个名为IMIM的新基准，专注于多实例场景。我们提出的IMD在常用评估基准上建立了新的最先进水平，而在IMIM上12%的显著提升表明我们的方法有效缓解了不对齐问题。

英文摘要：

Leveraging the vision foundation models has emerged as a mainstream paradigm that improves the performance of image feature matching. However, previous works have ignored the misalignment when introducing the foundation models into feature matching. The misalignment arises from the discrepancy between the foundation models focusing on single-image understanding and the cross-image understanding requirement of feature matching. Specifically, 1) the embeddings derived from commonly used foundation models exhibit discrepancies with the optimal embeddings required for feature matching; 2) lacking an effective mechanism to leverage the single-image understanding ability into cross-image understanding. A significant consequence of the misalignment is they struggle when addressing multi-instance feature matching problems. To address this, we introduce a simple but effective framework, called IMD (Image feature Matching with a pre-trained Diffusion model) with two parts: 1) Unlike the dominant solutions employing contrastive-learning based foundation models that emphasize global semantics, we integrate the generative-based diffusion models to effectively capture instance-level details. 2) We leverage the prompt mechanism in generative model as a natural tunnel, propose a novel cross-image interaction prompting module to facilitate bidirectional information interaction between image pairs. To more accurately measure the misalignment, we propose a new benchmark called IMIM, which focuses on multi-instance scenarios. Our proposed IMD establishes a new state-of-the-art in commonly evaluated benchmarks, and the superior improvement 12% in IMIM indicates our method efficiently mitigates the misalignment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。