SynHLMA:基于离散人类物体交互表示的合成手语操纵
SynHLMA:Synthesizing Hand Language Manipulation for Articulated Object with Discrete Human Object Interaction Representation
AI总结：
SynHLMA通过离散人类物体交互表示生成可变形物体的手语操纵序列，实现抓取生成、预测和插值任务。
摘要：
生成手抓取与语言指令相结合的抓取是广泛研究的主题，受益于具身AI和VR/AR应用。在转换到手对可变形物体的交互（HAOI）时，手抓取合成不仅需要物体功能，还需要沿物体变形的长期操纵序列。本文提出了一种新的HAOI序列生成框架SynHLMA，用于合成可变形物体的手语操纵。给定一个完整的点云可变形物体，我们利用离散的HAOI表示来建模每个手物体交互帧。同时，这些表示通过HAOI操纵语言模型进行训练，以在共享表示空间中对抓取过程与语言描述进行对齐。采用联合感知损失，确保手抓取遵循可变形物体关节的动态变化。通过这种方式，我们的SynHLMA实现了三种典型的可变形物体HAOI生成、预测和插值任务。我们在构建的HAOI-lang数据集上评估了SynHLMA，并实验结果表明其在手抓取序列生成方面优于最先进的方法。我们还展示了一个机器人抓取应用，该应用通过我们的SynHLMA提供的操纵序列实现了从模仿学习中执行灵巧抓取。我们的代码和数据集将公开发布。
英文摘要：
Generating hand grasps with language instructions is a widely studied topic that benefits from embodied AI and VR/AR applications. While transferring into hand articulatied object interaction (HAOI), the hand grasps synthesis requires not only object functionality but also long-term manipulation sequence along the object deformation. This paper proposes a novel HAOI sequence generation framework SynHLMA, to synthesize hand language manipulation for articulated objects. Given a complete point cloud of an articulated object, we utilize a discrete HAOI representation to model each hand object interaction frame. Along with the natural language embeddings, the representations are trained by an HAOI manipulation language model to align the grasping process with its language description in a shared representation space. A joint-aware loss is employed to ensure hand grasps follow the dynamic variations of articulated object joints. In this way, our SynHLMA achieves three typical hand manipulation tasks for articulated objects of HAOI generation, HAOI prediction and HAOI interpolation. We evaluate SynHLMA on our built HAOI-lang dataset and experimental results demonstrate the superior hand grasp sequence generation performance comparing with state-of-the-art. We also show a robotics grasp application that enables dexterous grasps execution from imitation learning using the manipulation sequence provided by our SynHLMA. Our codes and datasets will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
