Differentiable rendering is a very successful technique that applies to a Single-View 3D Reconstruction. Current renderers use losses based on pixels between a rendered image of some 3D reconstructed object and ground-truth images from given matched viewpoints to optimise parameters of the 3D shape.
These models require a rendering step, along with visibility handling and evaluation of the shading model. The main goal of this paper is to demonstrate that we can avoid these steps and still get reconstruction results as other state-of-the-art models that are equal or even better than existing category-specific reconstruction methods. First, we use the same CNN architecture for the prediction of a point cloud shape and pose prediction like the one used by Insafutdinov & Dosovitskiy. Secondly, we propose the novel effective loss function that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object's silhouette. Then we use Poisson Surface Reconstruction to transform the reconstructed point cloud into a 3D mesh. Finally, we perform a GAN-based texture mapping on a particular 3D mesh and produce a textured 3D mesh from a single 2D image. We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.
Few-shot NeRF by Adaptive Rendering Loss Regularization
Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）；Centre for Frontier AI Research（前沿人工智能研究中心）；A*STAR（新加坡科技研究局）
While recent learning based methods have been observed to be superior for several vision-related applications, their potential in generating artistic effects has not been explored much. One such interesting application is Shadow Art - a unique form of sculptural art where 2D shadows cast by a 3D sculpture produce artistic effects. In this work, we revisit shadow art using differentiable rendering based optimization frameworks to obtain the 3D sculpture from a set of shadow (binary) images and their corresponding projection information. Specifically, we discuss shape optimization through voxel …
Tsinghua University（清华大学）；University of Maryland（马里兰大学）
Differentiable renderers have been used successfully for unsupervised 3D structure learning from 2D images because they can bridge the gap between 3D and 2D. To optimize 3D shape parameters, current renderers rely on pixel-wise losses between rendered images of 3D reconstructions and ground truth images from corresponding viewpoints. Hence they require interpolation of the recovered 3D structure at each pixel, visibility handling, and optionally evaluating a shading model. In contrast, here we propose a Differentiable Renderer Without Rendering (DRWR) that omits these steps. DRWR only relies o…
The University of Hong Kong（香港大学）；Zhejiang University（浙江大学）；Apple（苹果公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Texas A&M University（德克萨斯农工大学）
This paper presents a Progressively-connected Light Field network (ProLiF), for the novel view synthesis of complex forward-facing scenes. ProLiF encodes a 4D light field, which allows rendering a large batch of rays in one training step for image- or patch-level losses. Directly learning a neural light field from images has difficulty in rendering multi-view consistent images due to its unawareness of the underlying 3D geometry. To address this problem, we propose a progressive training scheme and regularization losses to infer the underlying geometry during training, both of which enforce th…
Recently, great progress has been made in 3D deep learning with the emergence of deep neural networks specifically designed for 3D point clouds. These networks are often trained from scratch or from pre-trained models learned purely from point cloud data. Inspired by the success of deep learning in the image domain, we devise a novel pre-training technique for better model initialization by utilizing the multi-view rendering of the 3D data. Our pre-training is self-supervised by a local pixel/point level correspondence loss computed from perspective projection and a global image/point cloud le…
ProtoGS: Efficient and High-Quality Rendering with 3D Gaussian Prototypes
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Melbourne（墨尔本大学）；Alibaba Group（阿里巴巴集团）；The University of Sydney（悉尼大学）；Carnegie Mellon University（卡内基梅隆大学）
University of Science and Technology of China（中国科学技术大学）
In this paper, we propose HeadNeRF, a novel NeRF-based parametric head model that integrates the neural radiance field to the parametric representation of the human head. It can render high fidelity head images in real-time on modern GPUs, and supports directly controlling the generated images' rendering pose and various semantic attributes. Different from existing related parametric models, we use the neural radiance fields as a novel 3D proxy instead of the traditional 3D textured mesh, which makes that HeadNeRF is able to generate high fidelity images. However, the computationally expensive…
University of Maryland, College Park（马里兰大学帕克分校）；Amazon.com, Inc.（亚马逊公司）
We present FlexNeRF, a method for photorealistic freeviewpoint rendering of humans in motion from monocular videos. Our approach works well with sparse views, which is a challenging scenario when the subject is exhibiting fast/complex motions. We propose a novel approach which jointly optimizes a canonical time and pose configuration, with a pose-dependent motion field and pose-independent temporal deformations complementing each other. Thanks to our novel temporal and cyclic consistency constraints along with additional losses on intermediate representation such as segmentation, our approach …
Nanjing University of Science and Technology（南京理工大学）
Point clouds obtained from 3D sensors are usually sparse. Existing methods mainly focus on upsampling sparse point clouds in a supervised manner by using dense ground truth point clouds. In this paper, we propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images. Specifically, we first propose a neighbor expansion unit (NEU) to upsample the sparse point clouds, where the local geo…
Adaptive Time-step Training for Enhancing Spike-Based Neural Radiance Fields
School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）；Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）
Rendering Anywhere You See: Renderability Field-guided Gaussian Splatting
Laboratory of Artificial Intelligence Robotics（人工智能机器人实验室）；Politecnico di Milano（米兰理工学院）；Ningbo Institute of Materials Technology and Engineering（宁波材料技术与工程研究所）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
In this paper, we proposed a novel Style-based Point Generator with Adversarial Rendering (SpareNet) for point cloud completion. Firstly, we present the channel-attentive EdgeConv to fully exploit the local structures as well as the global shape in point features. Secondly, we observe that the concatenation manner used by vanilla foldings limits its potential of generating a complex and faithful shape. Enlightened by the success of StyleGAN, we regard the shape feature as style code that modulates the normalization layers during the folding, which considerably enhances its capability. Thirdly,…
GaRField++: Reinforced Gaussian Radiance Fields for Large-Scale 3D Scene Reconstruction
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；Department of Electrical and Computer Engineering, Clarkson University（克拉克森大学电气与计算机工程系）；Yunnan Key Laboratory of Intelligent Systems and Computing, Yunnan University（云南大学云南省智能系统与计算重点实验室）
We present a method for fast 3D reconstruction and real-time rendering of dynamic humans from monocular videos with accompanying parametric body fits. Our method can reconstruct a dynamic human in less than 3h using a single GPU, compared to recent state-of-the-art alternatives that take up to 72h. These speedups are obtained by using a lightweight deformation model solely based on linear blend skinning, and an efficient factorized volumetric representation for modeling the shape and color of the person in canonical pose. Moreover, we propose a novel local ray marching rendering which, by expl…
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and g…
Towards Photorealistic and Efficient Bokeh Rendering via Diffusion Framework
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；vivo BlueImage Lab, vivo Mobile Communication Co., Ltd.（vivo BlueImage实验室，vivo移动通信有限公司）；Shenzhen University of Advanced Technology（深圳大学）
TOGS: Gaussian Splatting with Temporal Opacity Offset for Real-Time 4D DSA Rendering
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Union Hospital, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属协和医院）
本文提出 TOGS，通过随时间变化的 Gaussian 不透明度偏移表、Smooth loss 和随机剪枝，实现稀疏视角下高质量、低存储的实时 4D DSA 渲染。
This paper proposes a novel approach for rendering a pre-trained Neural Radiance Field (NeRF) in real-time on resource-constrained devices. We introduce Re-ReND, a method enabling Real-time Rendering of NeRFs across Devices. Re-ReND is designed to achieve real-time performance by converting the NeRF into a representation that can be efficiently processed by standard graphics pipelines. The proposed method distills the NeRF by extracting the learned density into a mesh, while the learned color information is factorized into a set of matrices that represent the scene's light field. Factorization…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Learning neural radiance fields of a scene has recently allowed realistic novel view synthesis of the scene, but they are limited to synthesize images under the original fixed lighting condition. Therefore, they are not flexible for the eagerly desired tasks like relighting, scene editing and scene composition. To tackle this problem, several recent methods propose to disentangle reflectance and illumination from the radiance field. These methods can cope with solid objects with opaque surfaces but participating media are neglected. Also, they take into account only direct illumination or at m…
Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splatting
College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Baidu VIS（百度VIS）；Institute of Medical Technology, Peking University（北京大学医学技术研究所）
NTT Communication Science Laboratories, NTT Corporation（NTT株式会社NTT通信科学实验室）
Fully unsupervised 3D representation learning has gained attention owing to its advantages in data collection. A successful approach involves a viewpoint-aware approach that learns an image distribution based on generative models (e.g., generative adversarial networks (GANs)) while generating various view images based on 3D-aware models (e.g., neural radiance fields (NeRFs)). However, they require images with various views for training, and consequently, their application to datasets with few or limited viewpoints remains a challenge. As a complementary approach, an aperture rendering GAN (AR-…
Packet-Loss Robust 3D Gaussian Compression via Atomic Packaging and GNN-based Error Concealment
Central South University Changsha China；Malanshan Audio \& Video Laboratory Changsha China；Central South University；Malanshan Audio \& Video Laboratory
University of California, Berkeley（加州大学伯克利分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Even as deep neural networks (DNNs) have achieved remarkable success on vision-related tasks, their performance is brittle to transformations in the input. Of particular interest are semantic transformations that model changes that have a basis in the physical world, such as rotations, translations, changes in lighting or camera pose. In this paper, we show how differentiable rendering can be utilized to generate images that are informative, yet realistic, and which can be used to analyze DNN performance and improve its robustness through data augmentation. Given a differentiable renderer and …
GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflows
Shanghai Jiao Tong University（上海交通大学）；Xiaohongshu Inc.（小红书公司）；Hong Kong University of Science and Technology（香港科技大学）；Southeast University（东南大学）；South China University of Technology（华南理工大学）
