使用语义UV映射对三维人体进行纹理化仍然是一个挑战，因为难以获取合理展开的UV。尽管近期在利用大型文本到图像（T2I）模型监督多视角渲染的文本到三维生成方面取得了进展，但生成速度、文本一致性和纹理质量等问题依然存在，导致现有数据集中的数据稀缺。我们提出了TexDreamer，这是首个零样本多模态高保真三维人体纹理生成模型。利用一种高效的纹理适应微调策略，我们将大型T2I模型适应到语义UV结构，同时保留其原有的泛化能力。借助一个新颖的特征翻译器模块，训练后的模型能够在数秒内从文本或图像生成高保真三维人体纹理。此外，我们引入了ArTicuLated humAn textureS（ATLAS），这是最大的高分辨率（1024×1024）三维人体纹理数据集，包含50k个带有文本描述的高保真纹理。
英文摘要：
Texturing 3D humans with semantic UV maps remains a challenge due to the difficulty of acquiring reasonably unfolded UV. Despite recent text-to-3D advancements in supervising multi-view renderings using large text-to-image (T2I) models, issues persist with generation speed, text consistency, and texture quality, resulting in data scarcity among existing datasets. We present TexDreamer, the first zero-shot multimodal high-fidelity 3D human texture generation model. Utilizing an efficient texture adaptation finetuning strategy, we adapt large T2I model to a semantic UV structure while preserving its original generalization capability. Leveraging a novel feature translator module, the trained model is capable of generating high-fidelity 3D human textures from either text or image within seconds. Furthermore, we introduce ArTicuLated humAn textureS (ATLAS), the largest high-resolution (1024 X 1024) 3D human texture dataset which contains 50k high-fidelity textures with text descriptions.
Many methods have been proposed over the years to tackle the task of facial 3D geometry and texture recovery from a single image. Such methods often fail to provide high-fidelity texture without relying on 3D facial scans during training. In contrast, the complementary task of 3D facial generation has not received as much attention. As opposed to the 2D texture domain, where GANs have proven to produce highly realistic facial images, the more challenging 3D geometry domain has not yet caught up to the same levels of realism and diversity. In this paper, we propose a novel unified pipeline for …
AlignVTOFF: Texture-Spatial Feature Alignment for High-Fidelity Virtual Try-Off
Sino-European School of Technology, Shanghai University（上海大学 sino-欧洲技术学院）；National Demonstration Center for Experimental Engineering Training Education, Shanghai University（上海大学国家级实验工程培训教育示范中心）
HKUST（香港科技大学）；SenseTime Research（商汤科技研究院）；Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）
We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views while preserving specific details of the input image. High-fidelity 3D GAN inversion is inherently challenging due to the geometry-texture trade-off in 3D inversion, where overfitting to a single view input image often damages the estimated geometry during the latent optimization. To solve this challenge, we propose a novel pipeline that builds on the pseudo-multi-view estimation with visibility analysis. We keep the original textures for the visible parts and …
Imperial College London（帝国理工学院）；University of Middlesex（密德萨斯大学）
In the past few years, a lot of work has been done towards reconstructing the 3D facial structure from single images by capitalizing on the power of Deep Convolutional Neural Networks (DCNNs). In the most recent works, differentiable renderers were employed in order to learn the relationship between the facial identity features and the parameters of a 3D morphable model for shape and texture. The texture features either correspond to components of a linear texture space or are learned by auto-encoders directly from in-the-wild images. In all cases, the quality of the facial texture reconstruct…
A lot of work has been done towards reconstructing the 3D facial structure from single images by capitalizing on the power of Deep Convolutional Neural Networks (DCNNs). In the recent works, the texture features either correspond to components of a linear texture space or are learned by auto-encoders directly from in-the-wild images. In all cases, the quality of the facial texture reconstruction is still not capable of modeling facial texture with high-frequency details. In this paper, we take a radically different approach and harness the power of Generative Adversarial Networks (GANs) and DC…
STS-GAN: Can We Synthesize Solid Texture with High Fidelity from Arbitrary 2D Exemplar?
Shandong Provincial Key Laboratory of Preparation and Measurement of Building Materials（山东省建筑材料制备与测试技术重点实验室）；South China University of Technology（华南理工大学）；Shandong Provincial Key Laboratory of Network Based Intelligent Computing（山东省网络环境智能计算技术重点实验室）；Quan Cheng Laboratory（泉城实验室）；Shandong Qiuqi Analysis Instrument Co., Ltd.（山东求奇分析仪器有限公司）；University of Waterloo（滑铁卢大学）；University of Jinan（济南大学）
FaceRefiner: High-Fidelity Facial Texture Refinement with Differentiable Rendering-based Style Transfer
School of Informatics, Xiamen University（厦门大学信息学院）；China Mobile (Hangzhou) Information Technology Co., Ltd.（中国移动（杭州）信息技术有限公司）；School of Computing and Data Science, Xiamen University Malaysia（马来西亚厦门大学计算与数据科学学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
Ecole Centrale de Lyon（里昂中央理工学院）；Beihang University（北京航空航天大学）
Although much progress has been made recently in 3D face reconstruction, most previous work has been devoted to predicting accurate and fine-grained 3D shapes. In contrast, relatively little work has focused on generating high-fidelity face textures. Compared with the prosperity of photo-realistic 2D face image generation, high-fidelity 3D face texture generation has yet to be studied. In this paper, we proposed a novel UV map generation model that predicts the UV map from a single face image. The model consists of a UV sampler and a UV generator. By selectively sampling the input face image's…
Seoul National University（首尔大学）；Purdue University（普渡大学）；NAVER WEBTOON
Reconstructing 3D human faces in the wild with the 3D Morphable Model (3DMM) has become popular in recent years. While most prior work focuses on estimating more robust and accurate geometry, relatively little attention has been paid to improving the quality of the texture model. Meanwhile, with the advent of Generative Adversarial Networks (GANs), there has been great progress in reconstructing realistic 2D images. Recent work demonstrates that GANs trained with abundant high-quality UV maps can produce high-fidelity textures superior to those produced by existing methods. However, acquiring …
Tel Aviv University（特拉维夫大学）；Cornell Tech, Cornell University（康奈尔科技学院，康奈尔大学）
As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over the texture appearance. To encourage fidelity to the input condition, we introduce a novel differen…
Recent powerful vision classifiers are biased towards textures, while shape information is overlooked by the models. A simple attempt by augmenting training images using the artistic style transfer method, called Stylized ImageNet, can reduce the texture bias. However, Stylized ImageNet approach has two drawbacks in fidelity and diversity. First, the generated images show low image quality due to the significant semantic gap betweeen natural images and artistic paintings. Also, Stylized ImageNet training samples are pre-computed before training, resulting in showing the lack of diversity for e…
Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models
Central South University（中南大学）；Changsha University of Science and Technology（长沙理工大学）；Hunan University（湖南大学）；Peng Cheng Laboratory（鹏城实验室）；Beihang University（北京航空航天大学）；Wuhan University（武汉大学）
The ability to produce convincing textural details is essential for the fidelity of synthesized person images. However, existing methods typically follow a ``warping-based'' strategy that propagates appearance features through the same pathway used for pose transfer. However, most fine-grained features would be lost due to down-sampling, leading to over-smoothed clothes and missing details in the output images. In this paper we presents RATE-Net, a novel framework for synthesizing person images with sharp texture details. The proposed framework leverages an additional texture enhancing module …
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident. In our work, we aim to train performant 3D generative models that synthesize textured meshes which can be directly consumed by 3D rendering engines, thus immediately usable in downstream applications. Prior works on 3D generative modeling either lack geometric details, are limited in the mesh topology they can produce, typically do not support textures, or utilize neural renderers in t…
Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On
ETH Zurich（苏黎世联邦理工学院）；Alibaba XR Lab（阿里巴巴XR实验室）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Enhancing Texture Generation with High-Fidelity Using Advanced Texture Priors
MetaApp AI Research（MetaApp AI研究院）；Zhengzhou University（郑州大学）；Beijing Institute of Technology（北京理工大学）；Beijing University of Technology（北京工业大学）；Northeastern University（东北大学）
VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction
Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Beijing Academy of Blockchain and Edge Computing（北京区块链与边缘计算研究院）
DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation
School of Computer Science, Fudan University（复旦大学计算机科学学院）；HiDream.ai Inc.（HiDream.ai公司）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Apple（苹果公司）
Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views. Since the availability of textured 3D shapes remains very limited, learning a 3D-supervised data-driven method that predicts a texture based on the 3D input is very challenging. We thus propose Texturify, a GAN-based method that leverages a 3D shape dataset of an object class and learns to reproduce the distribution of appearances observed in real images by generating high-quality textures. In particular, our meth…
Consistent Zero-shot 3D Texture Synthesis Using Geometry-aware Diffusion and Temporal Video Models
Department of Image, Chung-Ang University（Chung-Ang大学图像系）；Graphics AI Lab, NCSOFT（NCSOFT图形AI实验室）；Department of Computer Science, University of Seoul（首尔大学计算机科学系）
High-Fidelity Facial Albedo Estimation via Texture Quantization
University of Technology Sydney（悉尼科技大学）；Shanghai Jiao Tong University（上海交通大学）；DeepGlint（旷视科技（北京））；Insightface；Zhejiang University（浙江大学）；Imperial College London（伦敦帝国学院）
