arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.00562cs.CV
收藏

Text2HOI:文本引导的手-物交互三维运动生成

Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction

UNIST(蔚山国家科学技术研究所);KETI(韩国电子技术研究院);Adobe Research(奥多比研究院)

Junuk Cha, Jihyeon Kim, Jae Shin Yoon, Seungryul Baek

AI总结:

本文提出Text2HOI,将文本引导的三维手-物交互生成分解为接触生成与运动生成,利用VAE和Transformer扩散模型并结合手部细化模块,实现更真实、多样且可泛化到未见物体的交互生成。

摘要:

本文提出了首个以文本为引导、用于生成三维手-物交互序列的工作。主要挑战来自标注数据的缺乏:现有的真值数据集在交互类型和物体类别上远不具备泛化能力,这阻碍了从文本提示出发对具有正确物理含义(例如接触和语义)的多样化三维手-物交互进行建模。为应对这一挑战,我们提出将交互生成任务分解为两个子任务:手-物接触生成和手-物运动生成。在接触生成方面,一个基于VAE的网络以文本和物体网格作为输入,生成交互过程中手部表面与物体表面之间的接触概率。该网络学习多样化物体中独立于物体类别的局部几何结构,因此可适用于一般物体。在运动生成方面,一个基于Transformer的扩散模型利用该三维接触图作为强先验,通过学习增强后的标注数据集,生成物理上合理的手-物运动,并将其表示为文本提示的函数;其中我们为许多现有的三维手部和物体运动数据标注了文本标签。最后,我们进一步引入一个手部细化模块,该模块最小化物体表面与手部关节之间的距离,以提高物体-手部接触的时间稳定性并抑制穿透伪影。在实验中,我们证明与其他基线方法相比,我们的方法能够生成更真实且更多样化的交互。我们还表明我们的方法适用于未见过的物体。我们将发布我们的模型和新标注的数据,作为未来研究的坚实基础。代码和数据可在以下地址获取:https://github.com/JunukCha/Text2HOI。

英文摘要:

This paper introduces the first text-guided work for generating the sequence of hand-object interaction in 3D. The main challenge arises from the lack of labeled data where existing ground-truth datasets are nowhere near generalizable in interaction type and object category, which inhibits the modeling of diverse 3D hand-object interaction with the correct physical implication (e.g., contacts and semantics) from text prompts. To address this challenge, we propose to decompose the interaction generation task into two subtasks: hand-object contact generation; and hand-object motion generation. For contact generation, a VAE-based network takes as input a text and an object mesh, and generates the probability of contacts between the surfaces of hands and the object during the interaction. The network learns a variety of local geometry structure of diverse objects that is independent of the objects' category, and thus, it is applicable to general objects. For motion generation, a Transformer-based diffusion model utilizes this 3D contact map as a strong prior for generating physically plausible hand-object motion as a function of text prompts by learning from the augmented labeled dataset; where we annotate text labels from many existing 3D hand and object motion data. Finally, we further introduce a hand refiner module that minimizes the distance between the object surface and hand joints to improve the temporal stability of the object-hand contacts and to suppress the penetration artifacts. In the experiments, we demonstrate that our method can generate more realistic and diverse interactions compared to other baseline methods. We also show that our method is applicable to unseen objects. We will release our model and newly labeled data as a strong foundation for future research. Codes and data are available in: https://github.com/JunukCha/Text2HOI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑