3D重建在现代摄影测量系统中扮演着日益重要的角色。传统的卫星或航空遥感（RS）平台能够为大规模地形和城市的3D重建提供必要的数据源。即使使用低空无人机（UAV，Unmanned Aerial Vehicles），在城市峡谷和室内场景等复杂情况下，由于相机帧之间频繁的跟踪失败和高昂的数据采集成本，3D重建仍具有挑战性。近年来，球面图像因能够通过一次相机曝光记录周围环境而被广泛使用。与视场角（FOV，Field of View）有限的透视图像相比，球面图像能够以完整的水平和垂直FOV覆盖整个场景，并促进这些复杂场景中的相机跟踪与数据采集。随着专业级和消费级球面相机的快速发展和广泛使用，球面图像在城市和室内场景的3D建模中展现出巨大潜力。然而，经典的3D重建流程无法直接用于球面图像。此外，目前很少有专为球面图像3D重建设计的软件包。因此，本研究在数据采集、特征检测与匹配、图像定向以及密集匹配方面，对球面图像3D重建的最新技术进行了全面调查，同时展示了有前景的应用并讨论了潜在的发展前景。我们期望本研究能为指导未来的研究提供有启发性的线索。
英文摘要：
3D reconstruction plays an increasingly important role in modern photogrammetric systems. Conventional satellite or aerial-based remote sensing (RS) platforms can provide the necessary data sources for the 3D reconstruction of large-scale landforms and cities. Even with low-altitude UAVs (Unmanned Aerial Vehicles), 3D reconstruction in complicated situations, such as urban canyons and indoor scenes, is challenging due to frequent tracking failures between camera frames and high data collection costs. Recently, spherical images have been extensively used due to the capability of recording surrounding environments from one camera exposure. In contrast to perspective images with limited FOV (Field of View), spherical images can cover the whole scene with full horizontal and vertical FOV and facilitate camera tracking and data acquisition in these complex scenes. With the rapid evolution and extensive use of professional and consumer-grade spherical cameras, spherical images show great potential for the 3D modeling of urban and indoor scenes. Classical 3D reconstruction pipelines, however, cannot be directly used for spherical images. Besides, there exist few software packages that are designed for the 3D reconstruction of spherical images. As a result, this research provides a thorough survey of the state-of-the-art for 3D reconstruction of spherical images in terms of data acquisition, feature detection and matching, image orientation, and dense matching as well as presenting promising applications and discussing potential prospects. We anticipate that this study offers insightful clues to direct future research.
Multi-view stereo (MVS) reconstruction is essential for creating 3D models. The approach involves applying epipolar rectification followed by dense matching for disparity estimation. However, existing approaches face challenges in applying dense matching for images with different viewpoints primarily due to large differences in object scale. In this paper, we propose a spherical model for epipolar rectification to minimize distortions caused by differences in principal rays. We evaluate the proposed approach using two aerial-based datasets consisting of multi-camera head systems. We show throu…
In this report, we proposed a 3D reconstruction method for the full-view fisheye camera. The camera we used is Ricoh Theta, which captures spherical images and has a wide field of view (FOV). The conventional stereo apporach based on perspective camera model cannot be directly applied and instead we used a spherical camera model to depict the relation between 3D point and its corresponding observation in the image. We implemented a system that can reconstruct the 3D scene using captures from two or more cameras. A GUI is also created to allow users to control the view perspective and obtain a …
In this paper, we describe a method to capture nearly entirely spherical (360 degree) depth information using two adjacent frames from a single spherical video with motion parallax. After illustrating a spherical depth information retrieval using two spherical cameras, we demonstrate monocular spherical stereo by using stabilized first-person video footage. Experiments demonstrated that the depth information was retrieved on up to 97% of the entire sphere in solid angle. At a speed of 30 km/h, we were able to estimate the depth of an object located over 30 m from the camera. We also reconstruc…
As the spherical object can be seen everywhere, we should extract the ellipse image accurately and fit it by implicit algebraic curve in order to finish the 3D reconstruction. In this paper, we propose a new ellipse fitting algorithm which only needs three points to fit the projection of spherical object and is different from the traditional algorithms that need at least five point. The fitting procedure is just similar as the estimation of Fundamental Matrix estimation by seven points, and the RANSAC algorithm has also been used to exclude the interference of noise and scattered points.
Spherical-GOF: Geometry-Aware Panoramic Gaussian Opacity Fields for 3D Scene Reconstruction
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
Neural Discrete Representation Learning for Sparse-View CBCT Reconstruction: From Algorithm Design to Prospective Multicenter Clinical Evaluation
Wuhan University（武汉大学）；ShanghaiTech University（上海科技大学）；Union Hospital, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属协和医院）；Zhongda Hospital, Medical School, Southeast University（东南大学医学院附属中大医院）；Tongji Hospital, Huazhong University of Science and Technology（华中科技大学同济医院）；University of Washington School of Medicine（华盛顿大学医学院）；Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院北京协和医院）；Wuhan Artificial Intelligence Computing Center（武汉人工智能计算中心）；Wuhan Supercomputing Center（武汉超算中心）；The First Affiliated Hospital of Zhengzhou University（郑州大学第一附属医院）
Given a single RGB panorama, the goal of 3D layout reconstruction is to estimate the room layout by predicting the corners, floor boundary, and ceiling boundary. A common approach has been to use standard convolutional networks to predict the corners and boundaries, followed by post-processing to generate the 3D layout. However, the space-varying distortions in panoramic images are not compatible with the translational equivariance property of standard convolutions, thus degrading performance. Instead, we propose to use spherical convolutions. The resulting network, which we call OmniLayout pe…
Northwestern University（西北大学）；Wormpex AI Research（万沛克斯人工智能研究院）
HDR reconstruction is an important task in computer vision with many industrial needs. The traditional approaches merge multiple exposure shots to generate HDRs that correspond to the physical quantity of illuminance of the scene. However, the tedious capturing process makes such multi-shot approaches inconvenient in practice. In contrast, recent single-shot methods predict a visually appealing HDR from a single LDR image through deep learning. But it is not clear whether the previously mentioned physical properties would still hold, without training the network to explicitly model them. In th…
Friedrich-Alexander University Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Siemens Healthcare GmbH（西门子医疗有限公司）
Purpose: To develop an algorithm for robust partial Fourier (PF) reconstruction applicable to diffusion-weighted (DW) images with non-smooth phase variations. Methods: Based on an unrolled proximal splitting algorithm, a neural network architecture is derived which alternates between data consistency operations and regularization implemented by recurrent convolutions. In order to exploit correlations, multiple repetitions of the same slice are jointly reconstructed under consideration of permutation-equivariance. The algorithm is trained on DW liver data of 60 volunteers and evaluated on retro…
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；INRIA Sophia Antipolis（法国国立信息学及自动化研究院索菲亚·安蒂波利斯分院）
Compressed Sensing (CS) takes advantage of signal sparsity or compressibility and allows superb signal reconstruction from relatively few measurements. Based on CS theory, a suitable dictionary for sparse representation of the signal is required. In diffusion MRI (dMRI), CS methods were proposed to reconstruct diffusion-weighted signal and the Ensemble Average Propagator (EAP), and there are two kinds of Dictionary Learning (DL) methods: 1) Discrete Representation DL (DR-DL), and 2) Continuous Representation DL (CR-DL). DR-DL is susceptible to numerical inaccuracy owing to interpolation and re…
Uncertainty-Aware Cortical Surface Reconstruction and Spherical Registration for Heterogeneous Brain MRI
Danish Research Centre for Magnetic Resonance, Department of Radiology and Nuclear Medicine, Copenhagen University Hospital – Amager and Hvidovre（丹麦磁共振研究中心，放射科与核医学科，哥本哈根大学医院-阿马格和维多弗）；Department of Health Technology, Technical University of Denmark（技术丹麦大学健康技术系）；Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital（阿蒂努拉·A·马丁诺斯生物医学成像中心，麻省总医院）；Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；Department of Radiology, Harvard Medical School（哈佛医学院放射学系）；Hawkes Institute, University College London（霍克斯研究所，伦敦大学学院）
A novel neural network architecture, known as DL-ESPIRiT, is proposed to reconstruct rapidly acquired cardiac MRI data without field-of-view limitations which are present in previously proposed deep learning-based reconstruction frameworks. Additionally, a novel convolutional neural network based on separable 3D convolutions is integrated into DL-ESPIRiT to more efficiently learn spatiotemporal priors for dynamic image reconstruction. The network is trained on fully-sampled 2D cardiac cine datasets collected from eleven healthy volunteers with IRB approval. DL-ESPIRiT is compared against a sta…
Deep Learning-based Accelerated MR Cholangiopancreatography without Fully-sampled Data
Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃尔朗根-纽伦堡大学）；Siemens Healthineers（西门子医疗）；New York University Grossman School of Medicine（纽约大学格罗斯曼医学院）
University of Minnesota（明尼苏达大学）；Center for Magnetic Resonance Research（磁共振研究中心）
Purpose: To develop a strategy for training a physics-guided MRI reconstruction neural network without a database of fully-sampled datasets. Theory and Methods: Self-supervised learning via data under-sampling (SSDU) for physics-guided deep learning (DL) reconstruction partitions available measurements into two disjoint sets, one of which is used in the data consistency units in the unrolled network and the other is used to define the loss for training. The proposed training without fully-sampled data is compared to fully-supervised training with ground-truth data, as well as conventional comp…
Phase-contrast magnetic resonance imaging (MRI) provides time-resolved quantification of blood flow dynamics that can aid clinical diagnosis. Long in vivo scan times due to repeated three-dimensional (3D) volume sampling over cardiac phases and breathing cycles necessitate accelerated imaging techniques that leverage data correlations. Standard compressed sensing reconstruction methods require tuning of hyperparameters and are computationally expensive, which diminishes the potential reduction of examination times. We propose an efficient model-based deep neural reconstruction network and eval…
Evaluation of Machine Learning Reconstruction Techniques for Accelerated Brain MRI Scans
Humber River Hospital（亨伯河医院）；Toronto Western Hospital, University of Toronto（多伦多大学多伦多西区医院）；Foqus Technologies Inc.（Foqus 技术公司）；Robarts Research Institute, Western University（西安大略大学罗巴茨研究所）
Amsterdam UMC（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；MS Center Amsterdam（阿姆斯特丹多发性硬化中心）；Amsterdam Neuroscience（阿姆斯特丹神经科学研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Leiden University（莱顿大学）；Delft University of Technology（代尔夫特理工大学）
Machine Learning methods can learn how to reconstruct Magnetic Resonance Images and thereby accelerate acquisition, which is of paramount importance to the clinical workflow. Physics-informed networks incorporate the forward model of accelerated MRI reconstruction in the learning process. With increasing network complexity, robustness is not ensured when reconstructing data unseen during training. We aim to embed data consistency (DC) in deep networks while balancing the degree of network complexity. While doing so, we will assess whether either explicit or implicit enforcement of DC in varyin…
