arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.13596cs.CV
收藏

VGGT-Segmentor: 几何增强的跨视角分割

VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation

Hangzhou International Innovation Institute of Beihang University(北航杭州国际创新研究院);Beihang University(北京航空航天大学)

Yulu Gao, Bohao Zhang, Zongheng Tang, Jitong Liao, Wenjun Wu, Si Liu

AI总结:

针对不同视角下的实例分割难题,提出VGGT-Segmentor框架,结合VGGT的几何建模与新型Union分割头,通过掩码提示融合、点引导预测和迭代细化实现像素精确分割,并采用单图像自监督训练,在Ego-Exo4D基准上达到新最优性能。

摘要:

跨不同自我中心和外部中心视图的实例级对象分割是视觉理解中的基本挑战,对于具身AI和远程协作应用至关重要。由于尺度、视角和遮挡的剧烈变化,直接像素级匹配变得不稳定,使得该任务异常困难。尽管像VGGT这样的最新几何感知模型为特征对齐提供了坚实基础,但我们发现,即使其内部对象级注意力保持一致,它们在密集预测任务中常常因显著的像素级投影漂移而失败。为弥合这一差距,我们引入了VGGT-Segmentor(VGGT-S),一个将鲁棒几何建模与像素精确语义分割统一的框架。VGGT-S利用VGGT强大的跨视图特征表示,并引入了一种新颖的Union分割头。该分割头分三个阶段运行:掩码提示融合、点引导预测和迭代掩码细化,有效地将高级特征对齐转化为精确的分割掩码。此外,我们提出了一种单图像自监督训练策略,消除了对配对标注的需求,并实现了强大的泛化能力。在Ego-Exo4D基准上,VGGT-S在Ego到Exo和Exo到Ego任务中分别实现了67.7%和68.0%的平均IoU,显著优于先前方法。值得注意的是,我们的无对应预训练模型超越了大多数全监督基线,证明了我们方法的有效性和可扩展性。代码公开于:https://github.com/buaa-colalab/VGGT-S。

英文摘要:

Instance-level object segmentation across disparate egocentric and exocentric views is a fundamental challenge in visual understanding, critical for applications in embodied AI and remote collaboration. This task is exceptionally difficult due to severe changes in scale, perspective, and occlusion, which destabilize direct pixel-level matching. While recent geometry-aware models like VGGT provide a strong foundation for feature alignment, we find they often fail at dense prediction tasks due to significant pixel-level projection drift, even when their internal object-level attention remains consistent. To bridge this gap, we introduce VGGT-Segmentor (VGGT-S), a framework that unifies robust geometric modeling with pixel-accurate semantic segmentation. VGGT-S leverages VGGT's powerful cross-view feature representation and introduces a novel Union Segmentation Head. This head operates in three stages: mask prompt fusion, point-guided prediction, and iterative mask refinement, effectively translating high-level feature alignment into a precise segmentation mask. Furthermore, we propose a single-image self-supervised training strategy that eliminates the need for paired annotations and enables strong generalization. On the Ego-Exo4D benchmark, VGGT-S sets a new state-of-the-art, achieving 67.7% and 68.0% average IoU for Ego to Exo and Exo to Ego tasks, respectively, significantly outperforming prior methods. Notably, our correspondence-free pretrained model surpasses most fully-supervised baselines, demonstrating the effectiveness and scalability of our approach. Code is publicly available at: https://github.com/buaa-colalab/VGGT-S.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑