Neural Radiance Field (NeRF) technology has made significant strides in creating novel viewpoints. However, its effectiveness is hampered when working with sparsely available views, often leading to performance dips due to overfitting. FreeNeRF attempts to overcome this limitation by integrating implicit geometry regularization, which incrementally improves both geometry and textures. Nonetheless, an initial low positional encoding bandwidth results in the exclusion of high-frequency elements. The quest for a holistic approach that simultaneously addresses overfitting and the preservation of high-frequency details remains ongoing. This study presents a novel feature-matching-based sparse geometry regularization module, enhanced by a spatially consistent geometry filtering mechanism and a frequency-guided geometric regularization strategy. This module excels at accurately identifying high-frequency keypoints, effectively preserving fine structural details. Through progressive refinement of geometry and textures across NeRF iterations, we unveil an effective few-shot neural rendering architecture, designated as SGCNeRF, for enhanced novel view synthesis. Our experiments demonstrate that SGCNeRF not only achieves superior geometry-consistent outcomes but also surpasses FreeNeRF, with improvements of 0.7 dB in PSNR on LLFF and DTU.
TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment
Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能交叉学科项目，首尔国立大学）；Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Global Technology Research, Samsung Electronics（三星电子全球技术研究）；AIIS, ASRI, INMC, ISRC, Seoul National University（AIIS、ASRI、INMC、ISRC、首尔国立大学）
We propose LookinGood^π, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts…
University of Macau（澳门大学）；Tencent AI Lab（腾讯AI Lab）
Image harmonization aims to modify the color of the composited region with respect to the specific background. Previous works model this task as a pixel-wise image-to-image translation using UNet family structures. However, the model size and computational cost limit the ability of their models on edge devices and higher-resolution images. To this end, we propose a novel spatial-separated curve rendering network(S$^2$CRNet) for efficient and high-resolution image harmonization for the first time. In S$^2$CRNet, we firstly extract the spatial-separated embeddings from the thumbnails of the mask…
Planning and Rendering in Concert: DeepFusion of Autoregressive Layouts and Diffusion for Visual Text Generation
University of Science and Technology of China（中国科学技术大学）；Central South University（中南大学）；Louisiana State University（路易斯安那州立大学）；Arizona State University（亚利桑那州立大学）；Sun Yat-sen University（中山大学）
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；College of Computer Science, Nankai University（南开大学计算机学院）
Tune-Your-Style: Intensity-tunable 3D Style Transfer with Gaussian Splatting
School of Electronic and Computer Engineering, Peking University, Shenzhen, China（电子与计算机工程学院，北京大学深圳校区）；Pengcheng Laboratory, Shenzhen, China（鹏城实验室）；AI for Science (AI4S)-Preferred Program, Peking University Shenzhen Graduate School, China（人工智能科学（AI4S）优选计划，北京大学深圳研究生院）；Department of Automation and BNRist, Tsinghua University, Beijing, China（自动化系和BNRist，清华大学，北京，中国）；Dalian University of Technology, China（大连理工大学）
University of Chinese Academy of Sciences（中国科学院大学）；Inception Institute of Artificial Intelligence（启元人工智能研究院）；UBTech Research（优必选研究院）；ShanghaiTech University（上海科技大学）；Southern University of Science and Technology（南方科技大学）
Generative Adversarial Networks (GANs) have the capability of synthesizing images, which have been successfully applied to medical image synthesis tasks. However, most of existing methods merely consider the global contextual information and ignore the fine foreground structures, e.g., vessel, skeleton, which may contain diagnostic indicators for medical image analysis. Inspired by human painting procedure, which is composed of stroking and color rendering steps, we propose a Sketching-rendering Unconditional Generative Adversarial Network (SkrGAN) to introduce a sketch prior constraint to gui…
RDG-GS: Relative Depth Guidance with Gaussian Splatting for Real-time Sparse-View 3D Rendering
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Biomedical Engineering and Instrument Science, Zhejiang University（浙江大学生物医学工程与仪器科学学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）
Video-to-video synthesis (vid2vid) aims for converting high-level semantic inputs to photorealistic videos. While existing vid2vid methods can achieve short-term temporal consistency, they fail to ensure the long-term one. This is because they lack knowledge of the 3D world being rendered and generate each frame only based on the past few frames. To address the limitation, we introduce a novel vid2vid framework that efficiently and effectively utilizes all past generated frames during rendering. This is achieved by condensing the 3D world rendered so far into a physically-grounded estimate of …
HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance
Sun Yat-sen University（中山大学）；Xi’an Jiaotong University（西安交通大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；United Arab Emirates University（阿联酋大学）
Variable Aperture Bokeh Rendering via Customized Focal Plane Guidance
School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）；School of Digital Industry, Jiangxi Normal University（江西师范大学数字产业学院）；School of Computer Science, National University of Defense Technology（国防科技大学计算机学院）
When Distillation Breaks Motion Control: Restoring Generative Trajectories for Fast Video Generators
Zhejiang University of Technology（浙江工业大学）；University of New South Wales (UNSW Sydney)（新南威尔士大学（悉尼））；University of Auckland（奥克兰大学）；Zhejiang University（浙江大学）
MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）；Migu Culture Technology Co., Ltd（咪咕文化科技有限公司）；vivo Mobile Communication (Hangzhou) Co., Ltd.（维沃移动通信（杭州）有限公司）；vivo Mobile Communication Co. Ltd（维沃移动通信有限公司）；School of Computer Science, University of Birmingham（伯明翰大学计算机学院）；School of Computer and Information, Hefei University of Technology（合肥工业大学计算机与信息学院）；South China University of Technology（华南理工大学）
See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Speech and Acoustic Laboratory, Joy Future Academy, Jingdong Corporation（语音与声学实验室、Joy Future Academy、京东公司）；Aarhus University（阿arhus大学）
Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the …
Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）
Existing state-of-the-art novel view synthesis methods rely on either fairly accurate 3D geometry estimation or sampling of the entire space for neural volumetric rendering, which limit the overall efficiency. In order to improve the rendering efficiency by reducing sampling points without sacrificing rendering quality, we propose to build a novel view synthesis framework based on learned MVS priors that enables general, fast and photo-realistic view synthesis simultaneously. Specifically, fewer but important points are sampled under the guidance of depth probability distributions extracted fr…
