Farm3D：通过蒸馏2D扩散模型学习关节式3D动物
Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion
AI总结：
本研究提出Farm3D方法，利用Stable Diffusion生成合成训练数据并将扩散模型作为分数监督，从零学习关节式物体的单目3D重建网络，可快速从单张图像输出可控3D资产，兼顾分析与合成应用。
摘要：
我们提出了Farm3D，一种仅依赖预训练2D扩散图像生成器提供的“免费”虚拟监督，学习关节式物体的类别特定3D重建器的方法。现有方法在给定某一物体类别的单视图图像集合时，能够学习一个单目网络，预测任意物体实例的3D形状、反照率、光照和视角。但这些方法严重依赖人工整理的干净训练数据，获取成本高昂。我们提出了一个利用Stable Diffusion等图像生成器生成足够干净、无需进一步人工整理的合成训练数据的框架，实现了此类重建网络的从零学习。此外，我们将扩散模型作为分数函数引入，以优化学习过程。其核心思路是：对重建的某些方面（如视角和光照）进行随机化，生成重建后3D物体的虚拟视图，再由2D网络评估生成图像的质量，从而为重建器提供反馈。与基于蒸馏、为每个文本提示生成单个3D资产的工作不同，我们的方法产出的是一个单目重建网络，它能在几秒内通过单次前向传播，从任意真实或生成的图像中输出可控的3D资产。我们的网络可用于分析任务（包括单目重建），也可用于合成任务，为视频游戏等实时应用生成关节式资产。
英文摘要：
We present Farm3D, a method for learning category-specific 3D reconstructors for articulated objects, relying solely on "free" virtual supervision from a pre-trained 2D diffusion-based image generator. Recent approaches can learn a monocular network that predicts the 3D shape, albedo, illumination, and viewpoint of any object occurrence, given a collection of single-view images of an object category. However, these approaches heavily rely on manually curated clean training data, which are expensive to obtain. We propose a framework that uses an image generator, such as Stable Diffusion, to generate synthetic training data that are sufficiently clean and do not require further manual curation, enabling the learning of such a reconstruction network from scratch. Additionally, we incorporate the diffusion model as a score to enhance the learning process. The idea involves randomizing certain aspects of the reconstruction, such as viewpoint and illumination, generating virtual views of the reconstructed 3D object, and allowing the 2D network to assess the quality of the resulting image, thus providing feedback to the reconstructor. Unlike work based on distillation, which produces a single 3D asset for each textual prompt, our approach yields a monocular reconstruction network capable of outputting a controllable 3D asset from any given image, whether real or generated, in a single forward pass in a matter of seconds. Our network can be used for analysis, including monocular reconstruction, or for synthesis, generating articulated assets for real-time applications such as video games.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
