生成高质量的基于物理的渲染（PBR）材质，对于在下游任务中实现真实感渲染十分重要，但由于材质与光照的效应相互交织，这一问题仍具挑战性。现有方法通过在 3D 生成流程中引入材质分解取得了突破，却往往将高光烘焙到反照率中，并忽略金属度和粗糙度的空间变化属性。本文提出 PBR3DGen，这是一种两阶段网格生成方法，可生成高质量 PBR 材质，并集成了新颖的多视图 PBR 材质估计模型与 3D PBR 网格重建模型。具体而言，PBR3DGen 利用视觉语言模型（VLM）引导多视图扩散，精确捕捉反光金属材质的空间分布和固有属性。此外，我们将依赖视图的光照感知条件作为像素感知先验，以增强空间变化的材质属性。进一步地，重建模型可重建带有 PBR 材质的高质量网格。实验结果表明，PBR3DGen 显著优于现有方法，在 PBR 估计和网格生成方面取得了新的最先进结果。更多结果与可视化可访问项目页面：https://pbr3dgen1218.github.io/。
英文摘要：
Generating high-quality physically based rendering (PBR) materials is important to achieve realistic rendering in the downstream tasks, yet it remains challenging due to the intertwined effects of materials and lighting. While existing methods have made breakthroughs by incorporating material decomposition in the 3D generation pipeline, they tend to bake highlights into albedo and ignore spatially varying properties of metallicity and roughness. In this work, we present PBR3DGen, a two-stage mesh generation method with high-quality PBR materials that integrates the novel multi-view PBR material estimation model and a 3D PBR mesh reconstruction model. Specifically, PBR3DGen leverages vision language models (VLM) to guide multi-view diffusion, precisely capturing the spatial distribution and inherent attributes of reflective-metalness material. Additionally, we incorporate view-dependent illumination-aware conditions as pixel-aware priors to enhance spatially varying material properties. Furthermore, our reconstruction model reconstructs high-quality mesh with PBR materials. Experimental results demonstrate that PBR3DGen significantly outperforms existing methods, achieving new state-of-the-art results for PBR estimation and mesh generation. More results and visualization can be found on our project page: https://pbr3dgen1218.github.io/.
MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
Helmholtz Zentrum Hereon（亥姆霍兹赫里恩研究中心）；Institute for Continuum and Material Mechanics, Hamburg University of Technology（汉堡工业大学连续介质与材料力学研究所）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Baidu VIS（百度VIS）；Institute of Medical Technology, Peking University（北京大学医学技术研究所）
MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery
Communication University of China（中国传媒大学）；Nankai University（南开大学）；Beihang University（北京航空航天大学）；East China Normal University（华东师范大学）；National University of Singapore（新加坡国立大学）
MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Tencent Hunyuan（腾讯文元）；The Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
PacTure: Efficient PBR Texture Generation on Packed Views with Visual Autoregressive Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息处理国家重点实验室，计算机学院，北京大学）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（视觉技术国家工程研究中心，计算机学院，北京大学）；The Hong Kong University of Science and Technology（香港科技大学）；Light Illusions；PKU-AI 2 Robotics Joint Lab of Embodied AI（北京大学人工智能2机器人联合实验室）
Zhejiang University（浙江大学）；Shanghai Innovation Institute（上海创新研究院）；Westlake University（西湖大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Fudan University（复旦大学）
ArchMap: Arch-Flattening and Knowledge-Guided Vision Language Model for Tooth Counting and Structured Dental Understanding
Xi'an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）；Zhejiang University of Technology（浙江工业大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generation
PCA Lab, VCIP, College of Computer Science, Nankai University（南开大学计算机学院VCIP实验室PCA Lab）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；Horizon Robotics（地平线机器人）；D-Robotics
Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering
University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；POSTECH（浦项科技大学）；Institute for Convergence Research and Education in Advanced Technology, Yonsei University（延世大学尖端技术融合研究与教育学院）
GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
