arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.03042cs.CV
收藏

AWOL:使用语言进行无合成分析

AWOL: Analysis WithOut synthesis using Language

IMATI-CNR(意大利国家研究委员会IMATI研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)

Silvia Zuffi, Michael J. Black

AI总结:

本文提出 AWOL,通过少量图文对学习视觉-语言模型潜空间到参数化三维模型参数空间的映射,实现由文本或图像生成未见的四足动物与三维树木。

摘要:

许多经典的参数化三维形状模型已经存在,但使用这类模型创建新颖形状需要具备关于其参数的专业知识。例如,设想使用程序化图形创建一种特定类型的树,或从统计形状模型创建一种新的动物。我们的核心思想是利用语言控制这些现有模型,以生成新颖形状。这涉及学习视觉-语言模型的潜空间与三维模型参数空间之间的映射,我们使用少量形状与文本配对数据来完成这一映射。我们的假设是,从语言到参数的映射使我们能够为训练期间从未见过的物体生成参数。如果语言与参数之间的映射足够平滑,那么语言中的插值或泛化就应当能够适当地转化为新颖的三维形状。我们在两种差异很大的参数化形状模型(四足动物和乔木)上测试了我们的方法。我们使用一个学习得到的四足动物统计形状模型,并表明可以利用文本生成训练期间不存在的新动物。特别是,我们展示了最先进的三维狗形状估计性能。本工作还构成了首个用于生成三维树木的语言驱动方法。最后,将图像嵌入到 CLIP 潜空间中,使我们能够直接从图像生成动物和树木。

英文摘要:

Many classical parametric 3D shape models exist, but creating novel shapes with such models requires expert knowledge of their parameters. For example, imagine creating a specific type of tree using procedural graphics or a new kind of animal from a statistical shape model. Our key idea is to leverage language to control such existing models to produce novel shapes. This involves learning a mapping between the latent space of a vision-language model and the parameter space of the 3D model, which we do using a small set of shape and text pairs. Our hypothesis is that mapping from language to parameters allows us to generate parameters for objects that were never seen during training. If the mapping between language and parameters is sufficiently smooth, then interpolation or generalization in language should translate appropriately into novel 3D shapes. We test our approach with two very different types of parametric shape models (quadrupeds and arboreal trees). We use a learned statistical shape model of quadrupeds and show that we can use text to generate new animals not present during training. In particular, we demonstrate state-of-the-art shape estimation of 3D dogs. This work also constitutes the first language-driven method for generating 3D trees. Finally, embedding images in the CLIP latent space enables us to generate animals and trees directly from images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑