arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.00944cs.CVcs.GR
收藏

利用3D透视几何约束增强扩散模型

Enhancing Diffusion Models with 3D Perspective Geometry Constraints

University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Sony AI(索尼人工智能);Yale University(耶鲁大学)

Rishi Upadhyay, Howard Zhang, Yunhao Ba, Ethan Yang, Blake Gella, Sicheng Jiang, Alex Wong, Achuta Kadambi

AI总结:

本文提出在扩散模型训练中加入3D透视几何约束,以提升生成图像的透视准确性与真实感,并改善下游单目深度估计模型的零样本迁移性能。

摘要:

尽管透视在艺术中是一个被充分研究的主题,但在图像中它通常被视为理所当然。然而,对于近期涌现的高质量图像合成方法(如潜在扩散模型)而言,透视准确性并不是一项明确要求。由于这些方法能够输出范围广泛的可能图像,合成图像很难遵循线性透视原理。我们在生成模型的训练过程中引入了一种新颖的几何约束,以强制实现透视准确性。我们表明,使用该约束训练的模型输出不仅看起来更逼真,还提升了在生成图像上训练的下游模型的性能。主观人体试验显示,在70%的情况下,参与者更偏好使用我们的约束训练的潜在扩散模型所生成的图像,而非Stable Diffusion V2模型生成的图像。在我们的图像上进行微调的SOTA单目深度估计模型(如DPT和PixelFormer),在KITTI测试集上进行零样本迁移时,其RMSE和SqRel分别比在真实图像上训练的原始模型最高提升7.03%和19.3%。

英文摘要:

While perspective is a well-studied topic in art, it is generally taken for granted in images. However, for the recent wave of high-quality image synthesis methods such as latent diffusion models, perspective accuracy is not an explicit requirement. Since these methods are capable of outputting a wide gamut of possible images, it is difficult for these synthesized images to adhere to the principles of linear perspective. We introduce a novel geometric constraint in the training process of generative models to enforce perspective accuracy. We show that outputs of models trained with this constraint both appear more realistic and improve performance of downstream models trained on generated images. Subjective human trials show that images generated with latent diffusion models trained with our constraint are preferred over images from the Stable Diffusion V2 model 70% of the time. SOTA monocular depth estimation models such as DPT and PixelFormer, fine-tuned on our images, outperform the original models trained on real images by up to 7.03% in RMSE and 19.3% in SqRel on the KITTI test set for zero-shot transfer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑