arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.00372cs.CV
收藏

HyperSDFusion:桥接语言与几何中的层次结构以增强3D文本到形状生成

HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation

Technical University of Munich(慕尼黑工业大学);Imperial College London(伦敦帝国学院);Beihang University(北京航空航天大学);Zhongguancun Laboratory(中关村实验室)

Zhiying Leng, Tolga Birdal, Xiaohui Liang, Federico Tombari

AI总结:

本文提出HyperSDFusion,首次利用双曲空间学习文本与3D形状的层次表示,并通过双分支扩散模型实现更优的文本到3D形状生成,在Text2Shape数据集上取得最先进结果。

摘要:

从文本生成3D形状是3D表示学习中的一项基础任务。文本-形状对呈现出层次结构,其中像“椅子”这样的通用文本涵盖了椅子的所有3D形状,而更详细的提示则指向更具体的形状。此外,文本和3D形状本质上都是层次结构。然而,现有的Text2Shape方法,如SDFusion,并未利用这一点。在这项工作中,我们提出了HyperSDFusion,这是一种双分支扩散模型,可根据给定文本生成3D形状。由于双曲空间适合处理层次数据,我们提出在双曲空间中学习文本和3D形状的层次表示。首先,我们引入一个双曲文本-图像编码器,在双曲空间中学习文本的序列和多模态层次特征。此外,我们设计了一个双曲文本-图卷积模块,在双曲空间中学习文本的层次特征。为了充分利用这些文本特征,我们引入双分支结构,将文本特征嵌入到3D特征空间中。最后,为了赋予生成的3D形状以层次结构,我们设计了一种双曲层次损失。我们的方法是首个探索用于文本到形状生成的双曲层次表示的方法。在现有的文本到形状配对数据集Text2Shape上的实验结果表明,我们取得了最先进的结果。我们在HyperSDFusion.github.io上发布了实现代码。

英文摘要:

3D shape generation from text is a fundamental task in 3D representation learning. The text-shape pairs exhibit a hierarchical structure, where a general text like ``chair" covers all 3D shapes of the chair, while more detailed prompts refer to more specific shapes. Furthermore, both text and 3D shapes are inherently hierarchical structures. However, existing Text2Shape methods, such as SDFusion, do not exploit that. In this work, we propose HyperSDFusion, a dual-branch diffusion model that generates 3D shapes from a given text. Since hyperbolic space is suitable for handling hierarchical data, we propose to learn the hierarchical representations of text and 3D shapes in hyperbolic space. First, we introduce a hyperbolic text-image encoder to learn the sequential and multi-modal hierarchical features of text in hyperbolic space. In addition, we design a hyperbolic text-graph convolution module to learn the hierarchical features of text in hyperbolic space. In order to fully utilize these text features, we introduce a dual-branch structure to embed text features in 3D feature space. At last, to endow the generated 3D shapes with a hierarchical structure, we devise a hyperbolic hierarchical loss. Our method is the first to explore the hyperbolic hierarchical representation for text-to-shape generation. Experimental results on the existing text-to-shape paired dataset, Text2Shape, achieved state-of-the-art results. We release our implementation under HyperSDFusion.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑