Novel view synthesis has made significant progress in the field of 3D computer vision. However, the rendering of view-consistent novel views from imperfect camera poses remains challenging. In this paper, we introduce a hybrid bundle-adjusting 3D Gaussians model that enables view-consistent rendering with pose optimization. This model jointly extract image-based and neural 3D representations to simultaneously generate view-consistent images and camera poses within forward-facing scenes. The effective of our model is demonstrated through extensive experiments conducted on both real and synthetic datasets. These experiments clearly illustrate that our model can effectively optimize neural scene representations while simultaneously resolving significant camera pose misalignments. The source code is available at https://github.com/Bistu3DV/hybridBA.
Recent works such as BARF and GARF can bundle adjust camera poses with neural radiance fields (NeRF) which is based on coordinate-MLPs. Despite the impressive results, these methods cannot be applied to Generalizable NeRFs (GeNeRFs) which require image feature extractions that are often based on more complicated 3D CNN or transformer architectures. In this work, we first analyze the difficulties of jointly optimizing camera poses with GeNeRFs, and then further propose our DBARF to tackle these issues. Our DBARF which bundle adjusts camera poses by taking a cost feature map as an implicit cost …
Xi’an Jiaotong University（西安交通大学）；Ant Group（蚂蚁集团）；Tencent AI Lab（腾讯人工智能实验室）
Neural Radiance Fields (NeRF) have achieved photorealistic novel views synthesis; however, the requirement of accurate camera poses limits its application. Despite analysis-by-synthesis extensions for jointly learning neural 3D representations and registering camera frames exist, they are susceptible to suboptimal solutions if poorly initialized. We propose L2G-NeRF, a Local-to-Global registration method for bundle-adjusting Neural Radiance Fields: first, a pixel-wise flexible alignment, followed by a frame-wise constrained parametric alignment. Pixel-wise local alignment is learned in an unsu…
FA-BARF: Frequency Adapted Bundle-Adjusting Neural Radiance Fields
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）；Department of Automation, Tsinghua University（清华大学自动化系）；Chair for Computer Aided Medical Procedures and Augmented Reality, Technical University of Munich（慕尼黑工业大学计算机辅助医疗程序与增强现实讲座）
Depth-Guided Bundle Sampling for Efficient Generalizable Neural Radiance Field Reconstruction
Key Laboratory of Media Audio and Video (Communication University of China), Ministry of Education（媒体音频与视频重点实验室（中国传媒大学），教育部）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）
针对可泛化神经辐射场渲染计算量大、采样冗余的问题，提出深度引导的束采样策略，通过分组采样和自适应分配采样点，在DTU数据集上实现1.27 dB PSNR提升和47% FPS增加，并达到2倍渲染加速。
Carnegie Mellon University（卡内基梅隆大学）；Massachusetts Institute of Technology（麻省理工学院）；The University of Adelaide（阿德莱德大学）
Neural Radiance Fields (NeRF) have recently gained a surge of interest within the computer vision community for its power to synthesize photorealistic novel views of real-world scenes. One limitation of NeRF, however, is its requirement of accurate camera poses to learn the scene representations. In this paper, we propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect (or even unknown) camera poses -- the joint problem of learning neural 3D representations and registering camera frames. We establish a theoretical connection to classical image alignment and show…
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）；Beijing Academy of Artificial Intelligence, BAAI（北京人工智能研究院，BAAI）；Fudan University（复旦大学）；Peking University（北京大学）
ShanghaiTech University（上海科技大学）；Stereye Intelligent Technology Co.,Ltd.（视辰智能科技有限公司）；Huazhong University of Science and Technology（华中科技大学）
Some of the most exciting experiences that Metaverse promises to offer, for instance, live interactions with virtual characters in virtual environments, require real-time photo-realistic rendering. 3D reconstruction approaches to rendering, active or passive, still require extensive cleanup work to fix the meshes or point clouds. In this paper, we present a neural volumography technique called neural volumetric video or NeuVV to support immersive, interactive, and spatial-temporal rendering of volumetric video contents with photo-realism and in real-time. The core of NeuVV is to efficiently en…
This paper proposes a novel stroke-based rendering (SBR) method that translates images into vivid oil paintings. Previous SBR techniques usually formulate the oil painting problem as pixel-wise approximation. Different from this technique route, we treat oil painting creation as an adaptive sampling problem. Firstly, we compute a probability density map based on the texture complexity of the input image. Then we use the Voronoi algorithm to sample a set of pixels as the stroke anchors. Next, we search and generate an individual oil stroke at each anchor. Finally, we place all the strokes on th…
EOGS++: Earth Observation Gaussian Splatting with Internal Camera Refinement and Direct Panchromatic Rendering
Universite Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli（巴黎萨克雷大学，法国国家科学研究中心，巴黎萨克雷高等师范学校，博雷利中心）；Politecnico di Torino（都灵理工大学）；AMIAD, Pôle Recherche（AMIAD，研究部）；Institut Universitaire de France（法兰西大学研究院）
GBR: Generative Bundle Refinement for High-fidelity Gaussian Splatting with Enhanced Mesh Reconstruction
College of future information technology, Fudan University（未来信息科技学院，复旦大学）；School of Biomedical Engineering, Tsinghua University（生物医学工程学院，清华大学）；Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University（电磁波信息科学重点实验室（MoE），复旦大学）；Department of Automation, Tsinghua University（自动化系，清华大学）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能重点实验室，人工智能研究院，上海交通大学）
Huazhong University of Science and Technology（华中科技大学）；Huawei（华为）
Neural Radiance Field (NeRF) and its variants have exhibited great success on representing 3D scenes and synthesizing photo-realistic novel views. However, they are generally based on the pinhole camera model and assume all-in-focus inputs. This limits their applicability as images captured from the real world often have finite depth-of-field (DoF). To mitigate this issue, we introduce DoF-NeRF, a novel neural rendering approach that can deal with shallow DoF inputs and can simulate DoF effect. In particular, it extends NeRF to simulate the aperture of lens following the principles of geometri…
DenseSplat: Densifying Gaussian Splatting SLAM with Neural Radiance Prior
Department of Computer Science, Dalian University of Technology（大连理工大学计算机科学系）；Department of Mechano-informatics, Information Science and Technology, The University of Tokyo（东京大学机械信息学系、信息科学与技术系）；Institute of Medical Robotics and Department of Automation, Shanghai Jiao Tong University（上海交通大学医学机器人研究所、自动化系）；Key Laboratory of System Control and Information Processing, Ministry of Education（教育部系统控制与信息处理重点实验室）
We present FaceVerse, a fine-grained 3D Neural Face Model, which is built from hybrid East Asian face datasets containing 60K fused RGB-D images and 2K high-fidelity 3D head scan models. A novel coarse-to-fine structure is proposed to take better advantage of our hybrid dataset. In the coarse module, we generate a base parametric model from large-scale RGB-D images, which is able to predict accurate rough 3D face models in different genders, ages, etc. Then in the fine module, a conditional StyleGAN architecture trained with high-fidelity scan models is introduced to enrich elaborate facial ge…
2S-UDF: A Novel Two-stage UDF Learning Method for Robust Non-watertight Model Reconstruction from Multi-view Images
State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Wuhan University（武汉大学）；NC State University（北卡罗来纳州立大学）
This paper presents a neural incremental Structure-from-Motion (SfM) approach, Level-S$^2$fM, which estimates the camera poses and scene geometry from a set of uncalibrated images by learning coordinate MLPs for the implicit surfaces and the radiance fields from the established keypoint correspondences. Our novel formulation poses some new challenges due to inevitable two-view and few-view configurations in the incremental SfM pipeline, which complicates the optimization of coordinate MLPs for volumetric neural rendering with unknown camera poses. Nevertheless, we demonstrate that the strong i…
Compact 3D Gaussian Splatting For Dense Visual SLAM
Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；The University of Tokyo（东京大学）；Harvard University（哈佛大学）；University of Cambridge（剑桥大学）
Swiss Federal Institute of Technology in Zürich（苏黎世联邦理工学院）；The University of Queensland（昆士兰大学）
Recently, groundbreaking results have been presented on open-vocabulary semantic image segmentation. Such methods segment each pixel in an image into arbitrary categories provided at run-time in the form of text prompts, as opposed to a fixed set of classes defined at training time. In this work, we present a zero-shot volumetric open-vocabulary semantic scene segmentation method. Our method builds on the insight that we can fuse image features from a vision-language model into a neural implicit representation. We show that the resulting feature field can be segmented into different classes by…
We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Rob…
6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model
Fondazione Istituto Italiano di Tecnologia (IIT)（意大利技术研究院基金会）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会）；Università di Trento（特伦托大学）；Durham University（杜伦大学）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Calgary（卡尔加里大学）
This manuscript provides a new framework for calibration of optical instruments, in particular mobile cameras, using large-scale circular black and white target fields. New methods were introduced for (i) matching targets between images; (ii) adjusting the systematic eccentricity error of target centers; and (iii) iteratively improving the calibration solution through a free-network self-calibrating bundle adjustment. It was observed that the proposed target matching effectively matched circular targets in 270 mobile phone images from a complete calibration laboratory with robustness to Type I…
NeVStereo: A NeRF-Driven NVS-Stereo Architecture for High-Fidelity 3D Tasks
University of Washington（华盛顿大学）；University of Southern California（南加州大学）；CUHK-Shenzhen（香港中文大学（深圳））；USC Institute for Creative Technologies（南加州大学创意技术研究所）
InstantSplat: Sparse-view Gaussian Splatting in Seconds
University of Texas at Austin（德克萨斯大学）；Nvidia Research（英伟达研究）；Xiamen University（厦门大学）；Georgia Institute of Technology（佐治亚理工学院）；Stanford University（斯坦福大学）；University of Southern California（南加州大学）
Blending Distributed NeRFs with Tri-stage Robust Pose Optimization
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Beijing Institute of Technology（北京理工大学）；Xi’an University of Architecture and Technology（西安建筑科技大学）；University of Southern California（南加州大学）
Robust and High-Fidelity 3D Gaussian Splatting: Fusing Pose Priors and Geometry Constraints for Texture-Deficient Outdoor Scenes
School of Mechatronical Engineering, Beijing Institute of Technology（机械电子工程学院，北京理工大学）；Beiing National Research Center for Information Science and Technology（北京国家信息科学与技术研究中心）；Qiyuan Lab（启元实验室）；Peking University（北京大学）
Toward Semantic Communication for Real-time Mobile 3D Reconstruction
School of Electrical and Electronic Engineering, North China Electric Power University（华北电力大学电气与电子工程学院）；James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）；School of Computer Science and Technology, China University of Petroleum (East China)（中国石油大学（华东）计算机科学与技术学院）
针对实时移动 3D 重建中几何估计对通信失真敏感的问题，提出语义通信框架，含语义收发器输出带置信度图的重建图像，引入置信度引导几何估计方法，仿真显示该框架能保持图像质量并提升姿态估计精度和 3D 结构一致性。
