通过逐像素梯度裁剪增强高分辨率3D生成
Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping
AI总结：
针对高分辨率3D生成中经冻结VAE编码器反向传播的像素梯度失控问题，提出PGC逐像素裁剪梯度幅度并保留纹理方向，以低成本提升现有模型的渲染质量。
摘要：
高分辨率3D物体生成仍然是一项具有挑战性的任务，主要原因是缺乏全面的带标注训练数据。近期的进展旨在通过利用在大量精选网络数据集上预训练的图像生成模型，并借助分数蒸馏采样（SDS）等知识迁移技术来克服这一限制。为高效满足高分辨率渲染需求，通常需要采用基于潜在表示的模型，例如潜在扩散模型（LDM）。在该框架中出现了一个重大挑战：为计算单个图像像素的梯度，必须将梯度从指定的潜在空间反向传播通过图像模型的冻结组件，例如LDM中使用的VAE编码器。然而，这一梯度传播路径从未得到优化，在训练期间始终处于不受控状态。我们发现，这些未受监管的梯度会损害3D模型从图像生成模型中获取纹理相关信息的能力，从而导致外观合成质量较差。为应对这一总体挑战，我们提出了一种名为逐像素梯度裁剪（PGC）的创新操作，旨在无缝集成到现有3D生成模型中，从而提升其合成质量。具体而言，我们通过高效裁剪逐像素梯度来控制随机梯度的幅度，同时保留关键的纹理相关梯度方向。尽管该方法简单且额外成本极低，但大量实验证明了我们的PGC在增强现有3D生成模型用于高分辨率物体渲染的性能方面的有效性。
英文摘要：
High-resolution 3D object generation remains a challenging task primarily due to the limited availability of comprehensive annotated training data. Recent advancements have aimed to overcome this constraint by harnessing image generative models, pretrained on extensive curated web datasets, using knowledge transfer techniques like Score Distillation Sampling (SDS). Efficiently addressing the requirements of high-resolution rendering often necessitates the adoption of latent representation-based models, such as the Latent Diffusion Model (LDM). In this framework, a significant challenge arises: To compute gradients for individual image pixels, it is necessary to backpropagate gradients from the designated latent space through the frozen components of the image model, such as the VAE encoder used within LDM. However, this gradient propagation pathway has never been optimized, remaining uncontrolled during training. We find that the unregulated gradients adversely affect the 3D model's capacity in acquiring texture-related information from the image generative model, leading to poor quality appearance synthesis. To address this overarching challenge, we propose an innovative operation termed Pixel-wise Gradient Clipping (PGC) designed for seamless integration into existing 3D generative models, thereby enhancing their synthesis quality. Specifically, we control the magnitude of stochastic gradients by clipping the pixel-wise gradients efficiently, while preserving crucial texture-related gradient directions. Despite this simplicity and minimal extra cost, extensive experiments demonstrate the efficacy of our PGC in enhancing the performance of existing 3D generative models for high-resolution object rendering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
