arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.24907cs.CVcs.AIcs.LG
收藏

理解多视图Transformer

Understanding Multi-View Transformers

TUM(慕尼黑技术大学);Claude Bernard University Lyon 1(里昂第一大学);University of Cambridge(剑桥大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)

Michal Stary, Julien Gaubil, Ayush Tewari, Vincent Sitzmann

AI总结:

针对多视图Transformer内部机制不明的问题,提出从残差连接探测可视化3D表征的方法,揭示DUSt3R变体的隐状态演化、层作用及与显式位姿方法的差异,证实其可优化对应关系,为模型改进提供支撑。

摘要:

诸如DUSt3R之类的多视图Transformer正以前馈方式解决3D任务,革新着3D视觉领域。然而,与以往基于优化的流水线不同,多视图Transformer的内部机制尚不明确。其黑箱特性使得仅靠数据规模扩展之外的进一步改进颇具挑战,也增加了其在安全与可靠性关键型应用中的使用难度。本文提出一种从多视图Transformer各层残差连接中探测并可视化3D表征的方法。我们以此研究了DUSt3R模型的一个变体,揭示了其隐状态在各模块间的演化过程、各独立层的作用,并阐明了它与具备更强显式全局位姿归纳偏置的方法之间的差异。最后,我们证实所研究的DUSt3R变体能够估计对应关系,并通过重建的几何结构对其进行优化。本次分析所用代码可在https://github.com/JulienGaubil/und3rstand 获取。

英文摘要:

Multi-view transformers such as DUSt3R are revolutionizing 3D vision by solving 3D tasks in a feed-forward manner. However, contrary to previous optimization-based pipelines, the inner mechanisms of multi-view transformers are unclear. Their black-box nature makes further improvements beyond data scaling challenging and complicates usage in safety- and reliability-critical applications. Here, we present an approach for probing and visualizing 3D representations from the residual connections of the multi-view transformers' layers. In this manner, we investigate a variant of the DUSt3R model, shedding light on the development of its latent state across blocks, the role of the individual layers, and suggest how it differs from methods with stronger inductive biases of explicit global pose. Finally, we show that the investigated variant of DUSt3R estimates correspondences that are refined with reconstructed geometry. The code used for the analysis is available at https://github.com/JulienGaubil/und3rstand .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑