arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.17827cs.CVcs.AIcs.GRcs.LG
收藏

DiffH2O:基于扩散模型的文本描述手部与物体交互合成

DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions

ETH(苏黎世联邦理工学院);Meta

Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, Bugra Tekin

AI总结:

本文提出DiffH2O方法,通过将任务分解为抓取与文本操控两阶段并使用独立扩散模型,结合紧凑姿态表示与双重引导方案,从文本和物体几何合成逼真的单/双手3D交互运动,优于基线方法。

摘要:

在3D中生成自然的手部与物体交互具有挑战性,因为生成的手部和物体运动需要具备物理合理性和语义意义。此外,可用手物交互数据集的有限规模阻碍了对未见物体的泛化。本文提出了一种名为DiffH2O的新方法,能够根据提供的文本提示和物体几何形状,合成逼真的单手或双手物体交互。该方法引入了三种技术以实现从有限数据中的有效学习。首先,我们将任务分解为抓取阶段和基于文本的操控阶段,并为每个阶段使用单独的扩散模型。在抓取阶段,模型仅生成手部运动,而在操控阶段,手部和物体姿态均被合成。其次,我们提出了一种紧凑表示,将手部和物体姿态紧密耦合,有助于生成逼真的手物交互。第三,我们提出了两种不同的引导方案以允许对生成运动进行更多控制:抓取引导和详细文本引导。抓取引导采用单一目标抓取姿态,并引导扩散模型在抓取阶段结束时达到该抓取,从而提供对抓取姿态的控制。给定该阶段的抓取运动,可以在操控阶段提示多种不同动作。对于文本引导,我们为GRAB数据集贡献了全面的文本描述,并表明它们使我们的方法能够对手物交互进行更细粒度的控制。我们的定量和定性评估表明,所提出的方法优于基线方法,并产生了自然的手物运动。

英文摘要:

Generating natural hand-object interactions in 3D is challenging as the resulting hand and object motions are expected to be physically plausible and semantically meaningful. Furthermore, generalization to unseen objects is hindered by the limited scale of available hand-object interaction datasets. In this paper, we propose a novel method, dubbed DiffH2O, which can synthesize realistic, one or two-handed object interactions from provided text prompts and geometry of the object. The method introduces three techniques that enable effective learning from limited data. First, we decompose the task into a grasping stage and an text-based manipulation stage and use separate diffusion models for each. In the grasping stage, the model only generates hand motions, whereas in the manipulation phase both hand and object poses are synthesized. Second, we propose a compact representation that tightly couples hand and object poses and helps in generating realistic hand-object interactions. Third, we propose two different guidance schemes to allow more control of the generated motions: grasp guidance and detailed textual guidance. Grasp guidance takes a single target grasping pose and guides the diffusion model to reach this grasp at the end of the grasping stage, which provides control over the grasping pose. Given a grasping motion from this stage, multiple different actions can be prompted in the manipulation phase. For the textual guidance, we contribute comprehensive text descriptions to the GRAB dataset and show that they enable our method to have more fine-grained control over hand-object interactions. Our quantitative and qualitative evaluation demonstrates that the proposed method outperforms baseline methods and leads to natural hand-object motions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑