FlowHOI: 基于流的语义引导的双手-物体交互生成用于灵巧机器人操作
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
AI总结：
FlowHOI通过语义引导的流匹配框架生成双手-物体交互序列，提升机器人操作的准确性和效率。
摘要：
最近的视觉-语言-动作（VLA）模型可以生成合理的末端执行器运动，但它们在长时间跨度和接触丰富的任务中往往失败，因为底层的双手-物体交互（HOI）结构没有被显式表示。一种无具体形态的交互表示，能够捕捉这种结构，将使操作行为更容易验证和在不同机器人之间转移。我们提出了FlowHOI，一个两阶段的流匹配框架，生成语义引导、时间上连贯的HOI序列，包括手姿、物体姿以及手-物体接触状态，条件于一个以自身为中心的观察、一个语言指令和一个3D高斯点散（3DGS）场景重建。我们分离了以几何为中心的抓取与以语义为中心的操作，将后者条件于紧凑的3D场景标记，并采用运动-文本对齐损失，使生成的交互在物理场景布局和语言指令上都得到语义引导。为了应对高保真HOI监督的稀缺性，我们引入了一种重建流程，从大规模的以自身为中心的视频中恢复对齐的手-物体轨迹和网格，从而获得一个HOI先验，以实现鲁棒的生成。在GRAB和HOT3D基准测试中，FlowHOI实现了最高的动作识别准确率，并且比最强的扩散基线高出1.7倍的物理模拟成功率，同时提供40倍的推理加速。我们进一步在四个灵巧操作任务上展示了真实机器人执行，证明了生成的HOI表示在真实机器人执行管道中的可行性。
英文摘要：
Recent vision-language-action (VLA) models can generate plausible end-effector motions, yet they often fail in long-horizon, contact-rich tasks because the underlying hand-object interaction (HOI) structure is not explicitly represented. An embodiment-agnostic interaction representation that captures this structure would make manipulation behaviors easier to validate and transfer across robots. We propose FlowHOI, a two-stage flow-matching framework that generates semantically grounded, temporally coherent HOI sequences, comprising hand poses, object poses, and hand-object contact states, conditioned on an egocentric observation, a language instruction, and a 3D Gaussian splatting (3DGS) scene reconstruction. We decouple geometry-centric grasping from semantics-centric manipulation, conditioning the latter on compact 3D scene tokens and employing a motion-text alignment loss to semantically ground the generated interactions in both the physical scene layout and the language instruction. To address the scarcity of high-fidelity HOI supervision, we introduce a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale egocentric videos, yielding an HOI prior for robust generation. Across the GRAB and HOT3D benchmarks, FlowHOI achieves the highest action recognition accuracy and a 1.7$\times$ higher physics simulation success rate than the strongest diffusion-based baseline, while delivering a 40$\times$ inference speedup. We further demonstrate real-robot execution on four dexterous manipulation tasks, illustrating the feasibility of retargeting generated HOI representations to real-robot execution pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
