Existing deep methods produce highly accurate 3D reconstructions in stereo and multiview stereo settings, i.e., when cameras are both internally and externally calibrated. Nevertheless, the challenge of simultaneous recovery of camera poses and 3D scene structure in multiview settings with deep networks is still outstanding. Inspired by projective factorization for Structure from Motion (SFM) and by deep matrix completion techniques, we propose a neural network architecture that, given a set of point tracks in multiple images of a static scene, recovers both the camera parameters and a (sparse) scene structure by minimizing an unsupervised reprojection loss. Our network architecture is designed to respect the structure of the problem: the sought output is equivariant to permutations of both cameras and scene points. Notably, our method does not require initialization of camera parameters or 3D point locations. We test our architecture in two setups: (1) single scene reconstruction and (2) learning from multiple scenes. Our experiments, conducted on a variety of datasets in both internally calibrated and uncalibrated settings, indicate that our method accurately recovers pose and structure, on par with classical state of the art methods. Additionally, we show that a pre-trained network can be used to reconstruct novel scenes using inexpensive fine-tuning with no loss of accuracy.
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）
Istituto Nazionale di Ottica del Consiglio Nazionale delle Ricerche (CNR-INO)（意大利国家光学研究所（国家研究委员会光学研究所））；University of Florence（佛罗伦萨大学）；European Laboratory for Non-Linear Spectroscopy (LENS)（欧洲非线性光谱实验室）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Imperial College London（帝国理工学院）；Qualcomm AI Research（高通人工智能研究院）
SpaER: Learning Spatio-temporal Equivariant Representations for Fetal Brain Motion Tracking
Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Northeastern University（东北大学）；Massachusetts Institute of Technology（麻省理工学院）；University of California Irvine（加州大学欧文分校）
