X-ray, known for its ability to reveal internal structures of objects, is expected to provide richer information for 3D reconstruction than visible light. Yet, existing neural radiance fields (NeRF) algorithms overlook this important nature of X-ray, leading to their limitations in capturing structural contents of imaged objects. In this paper, we propose a framework, Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF), for sparse-view X-ray 3D reconstruction. Firstly, we design a Line Segment-based Transformer (Lineformer) as the backbone of SAX-NeRF. Linefomer captures internal structures of objects in 3D space by modeling the dependencies within each line segment of an X-ray. Secondly, we present a Masked Local-Global (MLG) ray sampling strategy to extract contextual and geometric information in 2D projection. Plus, we collect a larger-scale dataset X3D covering wider X-ray applications. Experiments on X3D show that SAX-NeRF surpasses previous NeRF-based methods by 12.56 and 2.49 dB on novel view synthesis and CT reconstruction. Code, models, and data are released at https://github.com/caiyuanhao1998/SAX-NeRF
NerT-CA: Efficient Dynamic Reconstruction from Sparse-view X-ray Coronary Angiography
Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, The Netherlands（数学与计算机科学系，埃因霍温理工大学，埃因霍温，荷兰）；Philips Healthcare, Best, The Netherlands（飞利浦医疗，贝斯特，荷兰）；Department of Electrical Engineering, Eindhoven University of Technology, The Netherlands（电子工程系，埃因霍温理工大学，荷兰）
NeRF-CA: Dynamic Reconstruction of X-ray Coronary Angiography with Extremely Sparse-views
Department of Mathematics & Computer Science, Eindhoven University of Technology（数学与计算机科学系，埃因霍温理工大学）；Philips Healthcare（飞利普医疗）；Department of Electrical Engineering, Eindhoven University of Technology（电子工程系，埃因霍温理工大学）；ASML（ASML公司）
3DGR-CAR: Coronary artery reconstruction from ultra-sparse 2D X-ray views with a 3D Gaussians representation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC)（中国科学技术大学生命科学与医学部生物医学工程学院）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advance Research, USTC（中国科学技术大学苏州研究院医学影像、机器人、分析计算与学习中心）
We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific d…
In this paper, we proposed a new deep learning based dense monocular SLAM method. Compared to existing methods, the proposed framework constructs a dense 3D model via a sparse to dense mapping using learned surface normals. With single view learned depth estimation as prior for monocular visual odometry, we obtain both accurate positioning and high quality depth reconstruction. The depth and normal are predicted by a single network trained in a tightly coupled manner.Experimental results show that our method significantly improves the performance of visual tracking and depth prediction in comp…
NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images
Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究生院）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学计算机学院视觉技术国家工程研究中心）；Beijing University of Posts and Telecommunications（北京邮电大学）
Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction
HIT (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；NWPU（西北工业大学）；Beijing Institute of Control and Engineering（北京控制与工程研究所）；Technical University of Munich（慕尼黑工业大学）
PhysicsNeRF: Physics-Guided 3D Reconstruction from Sparse Views
Department of Electrical and Computer Engineering, Texas A\&M University, Doha, Qatar（电气与计算机工程系，德克萨斯A&M大学，多哈，卡塔尔）；College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar（科学与工程学院，哈马德·本·卡伊夫大学，多哈，卡塔尔）
PhysicsNeRF提出物理引导的稀疏视图三维重建框架，通过四个约束和紧凑架构，在8视图下达到21.4 dB PSNR，并分析泛化差距，实现物理一致的可泛化表示。
LGDWT-GS: Local and Global Discrete Wavelet-Regularized 3D Gaussian Splatting for Sparse-View Scene Reconstruction
National Institute of Standards and Technology（国家标准与技术研究院）；Department of Physics, Colorado State University（科罗拉多州立大学物理系）；Electrical Engineering Department, University of Colorado（科罗拉多大学电气工程系）
AD-GS: Anatomical Density-guided 3D Gaussian Splatting for Sparse-View Dental CBCT Reconstruction
Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University Third Hospital（北京大学第三医院）；National Institute of Informatics（信息学研究所）；Peking University（北京大学）
SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers
University of Pittsburgh（匹兹堡大学）；University of Arizona（亚利桑那大学）；Tongji University（同济大学）；University of Central Florida（中佛罗里达大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Northwestern University（西北大学）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）
Computed tomography is widely used to examine internal structures in a non-destructive manner. To obtain high-quality reconstructions, one typically has to acquire a densely sampled trajectory to avoid angular undersampling. However, many scenarios require a sparse-view measurement leading to streak-artifacts if unaccounted for. Current methods do not make full use of the domain-specific information, and hence fail to provide reliable reconstructions for highly undersampled data. We present a novel framework for sparse-view tomography by decoupling the reconstruction into two steps: First, we …
Sparse-view CT reconstruction is important in a wide range of applications due to limitations on cost, acquisition time, or dosage. However, traditional direct reconstruction methods such as filtered back-projection (FBP) lead to low-quality reconstructions in the sub-Nyquist regime. In contrast, deep neural networks (DNNs) can produce high-quality reconstructions from sparse and noisy data, e.g. through post-processing of FBP reconstructions, as can model-based iterative reconstruction (MBIR), albeit at a higher computational cost. In this paper, we introduce a direct-reconstruction DNN metho…
Filtered back projection (FBP) is a classical method for image reconstruction from sinogram CT data. FBP is computationally efficient but produces lower quality reconstructions than more sophisticated iterative methods, particularly when the number of views is lower than the number required by the Nyquist rate. In this paper, we use a deep convolutional neural network (CNN) to produce high-quality reconstructions directly from sinogram data. A primary novelty of our approach is that we first back project each view separately to form a stack of back projections and then feed this stack as input…
ShanghaiTech University（上海科技大学）；The University of Edinburgh（爱丁堡大学）；The University of Hong Kong（香港大学）；Texas A&M University（得克萨斯农工大学）
Cone beam computed tomography (CBCT) has been widely used in clinical practice, especially in dental clinics, while the radiation dose of X-rays when capturing has been a long concern in CBCT imaging. Several research works have been proposed to reconstruct high-quality CBCT images from sparse-view 2D projections, but the current state-of-the-arts suffer from artifacts and the lack of fine details. In this paper, we propose SNAF for sparse-view CBCT reconstruction by learning the neural attenuation fields, where we have invented a novel view augmentation strategy to overcome the challenges int…
ShanghaiTech University（上海科技大学）；Chinese Academy of Sciences（中国科学院）；Shanghai Institute of Microsystem and Information Technology（上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Photoacoustic (PA) imaging is a biomedical imaging modality capable of acquiring high-contrast images of optical absorption at depths much greater than traditional optical imaging techniques. However, practical instrumentation and geometry limit the number of available acoustic sensors surrounding the imaging target, which results in the sparsity of sensor data. Conventional PA image reconstruction methods give severe artifacts when they are applied directly to the sparse PA data. In this paper, we firstly propose to employ a novel signal processing method to make sparse PA raw data more suita…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Suzhou Institute of Intelligent Computing Technology, CAS（中国科学院苏州智能计算技术研究院）；Shenzhen Institutes of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Beijing Information Science and Technology University（北京信息科技大学）；University of Science and Technology of China（中国科学技术大学）
While Computed Tomography (CT) reconstruction from X-ray sinograms is necessary for clinical diagnosis, iodine radiation in the imaging process induces irreversible injury, thereby driving researchers to study sparse-view CT reconstruction, that is, recovering a high-quality CT image from a sparse set of sinogram views. Iterative models are proposed to alleviate the appeared artifacts in sparse-view CT images, but the computation cost is too expensive. Then deep-learning-based methods have gained prevalence due to the excellent performances and lower computation. However, these methods ignore …
MVMS-RCN: A Dual-Domain Unfolding CT Reconstruction with Multi-sparse-view and Multi-scale Refinement-correction
Xiangtan University（湘潭大学）；Zhejiang Normal University（浙江师范大学）；The University of Liverpool（利物浦大学）；University of Strathclyde（斯特拉斯克莱德大学）；Hunan National Applied Mathematics Center（湖南国家应用数学中心）
Recently, a number of approaches to low-dose computed tomography (CT) have been developed and deployed in commercialized CT scanners. Tube current reduction is perhaps the most actively explored technology with advanced image reconstruction algorithms. Sparse data sampling is another viable option to the low-dose CT, and sparse-view CT has been particularly of interest among the researchers in CT community. Since analytic image reconstruction algorithms would lead to severe image artifacts, various iterative algorithms have been developed for reconstructing images from sparsely view-sampled pr…
Nanchang University（南昌大学）；Gannan Normal University（赣南师范大学）
The radiation dose in computed tomography (CT) examinations is harmful for patients but can be significantly reduced by intuitively decreasing the number of projection views. Reducing projection views usually leads to severe aliasing artifacts in reconstructed images. Previous deep learning (DL) techniques with sparse-view data require sparse-view/full-view CT image pairs to train the network with supervised manners. When the number of projection view changes, the DL network should be retrained with updated sparse-view/full-view CT image pairs. To relieve this limitation, we present a fully un…
This paper proposes a novel and fast self-supervised solution for sparse-view CBCT reconstruction (Cone Beam Computed Tomography) that requires no external training data. Specifically, the desired attenuation coefficients are represented as a continuous function of 3D spatial coordinates, parameterized by a fully-connected deep neural network. We synthesize projections discretely and train the network by minimizing the error between real and synthesized projections. A learning-based encoder entailing hash coding is adopted to help the network capture high-frequency details. This encoder outper…
DeepSparse: A Foundation Model for Sparse-View CBCT Reconstruction
Department of Electronic and Computer Engineering, the Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；School of Cyber Science and Engineering, Sichuan University（网络科学与工程学院，四川大学）
In this paper, the Primal-Dual UNet for sparse view CT reconstruction is modified to be applicable to cone beam projections and perform reconstructions of entire volumes instead of slices. Experiments show that the PSNR of the proposed method is increased by 10dB compared to the direct FDK reconstruction and almost 3dB compared to the modified original Primal-Dual Network when using only 23 projections. The presented network is not optimized wrt. memory consumption or hyperparameters but merely serves as a proof of concept and is limited to low resolution projections and volumes.
Deep learning affords enormous opportunities to augment the armamentarium of biomedical imaging, albeit its design and implementation have potential flaws. Fundamentally, most deep learning models are driven entirely by data without consideration of any prior knowledge, which dramatically increases the complexity of neural networks and limits the application scope and model generalizability. Here we establish a geometry-informed deep learning framework for ultra-sparse 3D tomographic image reconstruction. We introduce a novel mechanism for integrating geometric priors of the imaging system. We…
A growing number of applications require the reconstructionof 3D objects from a very small number of views. In this research, we consider the problem of reconstructing a 3D object from only 4 Flash X-ray CT views taken during the impact of a Kolsky bar. For such ultra-sparse view datasets, even model-based iterative reconstruction (MBIR) methods produce poor quality results. In this paper, we present a framework based on a generalization of Plug-and-Play, known as Multi-Agent Consensus Equilibrium (MACE), for incorporating complex and nonlinear prior information into ultra-sparse CT reconstruc…
Sparse representation using over-complete dictionaries have shown to produce good quality results in various image processing tasks. Dictionary learning algorithms have made it possible to engineer data adaptive dictionaries which have promising applications in image compression and image enhancement. The most common sparse dictionary learning algorithms use the techniques of matching pursuit and K-SVD iteratively for sparse coding and dictionary learning respectively. While this technique produces good results, it requires a large number of iterations to converge to an optimal solution. In th…
Traditional model-based image reconstruction (MBIR) methods combine forward and noise models with simple object priors. Recent application of deep learning methods for image reconstruction provides a successful data-driven approach to addressing the challenges when reconstructing images with measurement undersampling or various types of noise. In this work, we propose a hybrid supervised-unsupervised learning framework for X-ray computed tomography (CT) image reconstruction. The proposed learning formulation leverages both sparsity or unsupervised learning-based priors and neural network recon…
