arXiv 2403.078072024-03-12cs.CV

StyleGaussian：基于高斯泼溅的即时3D风格迁移

StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting

Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；UCAS-Terminus AI Lab, UCAS（中国科学院大学- Terminus AI实验室）

Kunhao Liu, Fangneng Zhan, Muyu Xu, Christian Theobalt, Ling Shao, Shijian Lu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出StyleGaussian 3D风格迁移技术，基于3DGS搭配高效特征渲染策略与K近邻3D CNN，实现10fps的即时风格迁移，同时兼顾风格质量、实时渲染与严格多视角一致性。

摘要：

我们提出了StyleGaussian，这是一种新颖的3D风格迁移技术，能够以10帧每秒（fps）的速度将任意图像的风格即时迁移到3D场景中。借助3D Gaussian Splatting（3D高斯泼溅，3DGS），StyleGaussian在实现风格迁移的同时，并未牺牲其实时渲染能力与多视角一致性。它通过三个步骤实现即时风格迁移：嵌入、迁移和解码。首先，将2D VGG场景特征嵌入到已重建的3D高斯中；接着，根据参考风格图像对嵌入的特征进行变换；最后，将变换后的特征解码为风格化的RGB图像。StyleGaussian有两项新颖设计。第一项是高效的特征渲染策略：在嵌入VGG特征时，先渲染低维特征，再将其映射到高维特征。该策略大幅降低了内存消耗，使3DGS能够渲染高内存占用的高维特征。第二项是基于K近邻的3D CNN，它作为风格化特征的解码器，消除了会损害严格多视角一致性的2D CNN操作。大量实验表明，StyleGaussian实现了即时3D风格化，在保持实时渲染与严格多视角一致性的同时，具备出色的风格化质量。项目页面：https://kunhao-liu.github.io/StyleGaussian/

英文摘要：

We introduce StyleGaussian, a novel 3D style transfer technique that allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps). Leveraging 3D Gaussian Splatting (3DGS), StyleGaussian achieves style transfer without compromising its real-time rendering ability and multi-view consistency. It achieves instant style transfer with three steps: embedding, transfer, and decoding. Initially, 2D VGG scene features are embedded into reconstructed 3D Gaussians. Next, the embedded features are transformed according to a reference style image. Finally, the transformed features are decoded into the stylized RGB. StyleGaussian has two novel designs. The first is an efficient feature rendering strategy that first renders low-dimensional features and then maps them into high-dimensional features while embedding VGG features. It cuts the memory consumption significantly and enables 3DGS to render the high-dimensional memory-intensive features. The second is a K-nearest-neighbor-based 3D CNN. Working as the decoder for the stylized features, it eliminates the 2D CNN operations that compromise strict multi-view consistency. Extensive experiments show that StyleGaussian achieves instant 3D stylization with superior stylization quality while preserving real-time rendering and strict multi-view consistency. Project page: https://kunhao-liu.github.io/StyleGaussian/

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。