arXiv 2508.052052025-08-07cs.CV

EndoMatcher：通过多域预训练实现机器人辅助手术的通用内窥镜图像匹配器

EndoMatcher: Generalizable Endoscopic Image Matcher via Multi-Domain Pre-training for Robot-Assisted Surgery

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院呼吸与危重症医学科）；Hong Kong Institute of Science & Innovation（香港创新科技署）；Centre of AI and Robotics, Hong Kong Institute of Science & Innovation（人工智能与机器人中心，香港创新科技署）

Bingyu Yang, Qingyao Tian, Yimeng Geng, Huai Liao, Xinyan Huang, Jiebo Luo, Hongbin Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对内窥镜图像匹配中视觉条件差和数据稀缺的问题，提出基于多域预训练的EndoMatcher，通过双分支ViT和渐进多目标训练，在零样本下显著提升匹配准确率和数量。

摘要：

在机器人辅助任务中，内窥镜图像的通用密集特征匹配对于三维重建、导航和手术场景理解至关重要。然而，由于困难的视觉条件（如弱纹理、大视角变化）和标注数据的稀缺，这仍然是一个挑战。为了解决这些挑战，我们提出了EndoMatcher，一种通过大规模、多域数据预训练实现的通用内窥镜图像匹配器。为了应对困难的视觉条件，EndoMatcher采用双分支视觉Transformer提取多尺度特征，并通过双交互模块增强鲁棒对应学习。为了克服数据稀缺并提高域多样性，我们构建了Endo-Mix6，这是首个用于内窥镜匹配的多域数据集。Endo-Mix6包含约120万对真实和合成图像对，涵盖六个域，对应标签通过运动恢复结构和模拟变换生成。Endo-Mix6的多样性和规模由于数据集大小、分布偏移和误差不平衡的显著差异，引入了训练稳定性的新挑战。为了解决这些问题，采用了一种渐进式多目标训练策略，以促进平衡学习并提高跨域的表示质量。这使得EndoMatcher能够以零样本方式泛化到未见过的器官和成像条件。广泛的零样本匹配实验表明，与最先进方法相比，EndoMatcher在Hamlyn和Bladder数据集上分别将内点匹配数量增加了140.69%和201.43%，并在Gastro-Matching数据集上将匹配方向预测准确率（MDPA）提高了9.40%，在具有挑战性的内窥镜条件下实现了密集且准确的匹配。代码公开于https://github.com/Beryl2000/EndoMatcher。

英文摘要：

Generalizable dense feature matching in endoscopic images is crucial for robot-assisted tasks, including 3D reconstruction, navigation, and surgical scene understanding. Yet, it remains a challenge due to difficult visual conditions (e.g., weak textures, large viewpoint variations) and a scarcity of annotated data. To address these challenges, we propose EndoMatcher, a generalizable endoscopic image matcher via large-scale, multi-domain data pre-training. To address difficult visual conditions, EndoMatcher employs a two-branch Vision Transformer to extract multi-scale features, enhanced by dual interaction blocks for robust correspondence learning. To overcome data scarcity and improve domain diversity, we construct Endo-Mix6, the first multi-domain dataset for endoscopic matching. Endo-Mix6 consists of approximately 1.2M real and synthetic image pairs across six domains, with correspondence labels generated using Structure-from-Motion and simulated transformations. The diversity and scale of Endo-Mix6 introduce new challenges in training stability due to significant variations in dataset sizes, distribution shifts, and error imbalance. To address them, a progressive multi-objective training strategy is employed to promote balanced learning and improve representation quality across domains. This enables EndoMatcher to generalize across unseen organs and imaging conditions in a zero-shot fashion. Extensive zero-shot matching experiments demonstrate that EndoMatcher increases the number of inlier matches by 140.69% and 201.43% on the Hamlyn and Bladder datasets over state-of-the-art methods, respectively, and improves the Matching Direction Prediction Accuracy (MDPA) by 9.40% on the Gastro-Matching dataset, achieving dense and accurate matching under challenging endoscopic conditions. The code is publicly available at https://github.com/Beryl2000/EndoMatcher.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。